Everything has changed. Beacon has fallen, Ozpin is gone, and those responsible have vanished without a trace. With their time studying at Beacon Academy ended abruptly, Team Ruby has separated, each of them following their own paths. Welcome back to Ruby Recap. I am Phoenix Knight, and today we will be covering the events of Ruby Vol. 4 and 5. In the Land of Darkness, Grimm spawn from black pools as Emerald and Mercury watch in disgust. It is here that Salem resides in Evernight Castle, and she meets with her lieutenants. The arrogant Dr. Arthur Watts, the stoic Hazel Reinhardt, the maniacal Tyrion Callows, and the injured Cinder Fall. After the battle at Beacon Tower, Cinder was left barely able to speak, and was now missing her left eye. Watts and Tyrion belittle Cinder for her loss as Hazel remarks that they have dealt with the SilverEyed Warriors before. Salem contradicts Watts' criticisms and praises Cinder for taking the Fall Maiden powers and killing Ozpin. She then orders Cinder to stay close, continuing her treatment, sending Watts to meet with her informant in Mistral, Hazel to meet with the leader of the White Fang alongside Adam Tauros, and Tyrion is sent to capture Ruby alive. Elsewhere in Remnant, a young boy awakens and begins his daily work on his farm. In a forest on the outskirts of Mistral, Ruby, Nora, Jean, and Ren, now collectively known as Team Ranger, battle against a Geist Grimm, a monster that can possess inanimate objects. It had been terrorizing a nearby town, and as payment for defeating it, Jean's equipment is upgraded using the metal that had once belonged to Pyrrha. Communication in Remnant is reliant on a series of relay towers, called the Cross Continental Transmit System. It consists of four primary towers, one located in each kingdom, and a number of smaller relay towers scattered across Remnant, to allow individuals to communicate and share information through radio waves. Due to the destruction of Beacon Tower, one of the four primary relay towers, global communications have been cut off, making it much more difficult for the smaller villages scattered between the kingdoms to call for help when needed. The next town they arrive at, Shion, is much less lively as it is still burning after being attacked by bandits and the Grimm that were drawn by the panic. Only a single injured huntsman remained barely alive, dying soon after Team Ranger had arrived, and leaving a sinking feeling in their hearts as Ren and Nora spot a large hoofprint on the ground. That night, Ruby dreams of Pyrrha once again, and awakens to find Jean as he trains to a recording of Pyrrha he has on his scroll with a heavy heart. Back in the Land of Darkness, Cinder is struggling as Salem is telling her to make something fear her instead of the other way around. They are then interrupted by a Seer Grimm, a being that Salem uses for communication. As it approaches, Salem turns to Cinder and asks again if she did kill Ozpin, to which Cinder chokes out a reply. Salem then orders the Seer to increase their forces at Beacon. She's sure that the relic is there. Meanwhile, the young farm boy, Oscar Pine, finished with his tasks for the day begins to hear a voice in his head, one that we are all too familiar with. As Team Ranger traveled, Crow followed, eliminating many of the Grimm they might encounter. Upon arriving at the next town, Hajibana, he is met by his sister Raven Bronwyn, who wants to know if Salem has claimed Ozpin's relic, to which Crow chastises Raven for not being there for Yang after she lost her arm, and for leading the Bandit tribe that attacked the village of Shion, a tribe that Crow had left behind. He asks Raven if she knows where the Spring Maiden is, and she refuses to answer, leaving through a portal once again. A short time after Crow and Team Ranger had left, another person arrives at the bar to ask for them. Tyrion is close in pursuit. As Team Ranger arrives at another town, named Oni Yuri, that stands completely abandoned, a town that Ren believes was destroyed by a single Grimm, Tyrion catches up with them, and after they refuse to hand Ruby over, he attacks. During the fight, Tyrion easily outmatches all of Team Ranger, and it is revealed that he is a Scorpion Faunus. As he is about to pierce Ruby with his Stinger, Crow arrives to save the day. As Crow realizes that Tyrion works for Salem, he instructs Team Ranger to stay back, as he is too dangerous. The fight drags on, and Crow and Tyrion's aura are both drained, prompting Ruby to jump back into the fight. As she does so, Crow was forced to save her from a piece of falling debris, giving Tyrion an opening to slash Crow with his Stinger. Ruby then severs Tyrion's extended tail, forcing him to flee as Team Ranger turned to the injured Crow for an explanation. Back on the farm, Ospin continues to talk to Oscar, telling him that he is not insane and that their souls have become linked. He stresses the importance of getting both of them to Haven Academy. Later that evening, Crow reveals to Team Ranger the truth of the Maidens and Salem's overall goal. Now, according to Ospin, there is one other fairy tale that has some truth within it. The tale of the two brothers, gods holding sway over the powers of creation and destruction. The older brother favoring creation, bringing life into the world, while the younger brother favored destruction, bringing to ruin all that the older brother had created, eventually leading him to create the Grim, creatures that know only destruction. In time, the brothers did make peace with one another by coming together and making one final creation, the human race, beings knowing both creation and destruction, possessing the knowledge to use it and having the choice to decide which path to follow, each of these four ideals becoming physical form in the relics. These relics are safeguarded under the Four Huntsmen Academies, and it is Salem's goal to possess all of them. Bewildered at what they had just heard, Ruby then asks Crow why he didn't travel with them in the first place, and he tells them of his semblance, misfortune, the ability to bring bad luck to those around him, something that he can't fully control, making it difficult for him to be close to others, especially those that he wishes to protect. Crow then ends the discussion, and the following morning, Team Ranger awakes to find Crow coughing and his wound oozing a purple substance before he falls unconscious. As Ruby and Jon carry Crow on a stretcher, the team splits to increase their chances of finding aid. Ren and Nora head through a mountain pass as Ruby and Jon head to the supposedly destroyed village of Kuro Yuri, as they leave Ruby and Jon pass over the same hoof print seen in Shion Village. Meanwhile, Oscar had made up his mind to accept Ozpin's request and headed for Mistral. While on his way to the train station, he has a brief encounter with Hazel, who actually assists him, not knowing of the passenger Oscar carries inside himself. The village of Kuro Yuri. In a time long past, this is where Ren grew up, living with his parents.  Later that evening, he is awoken by his mother in a panic, only to fall unconscious again and wake up in his father's arms. He tells Ren that he must be brave as Ren's father turns to face the monster that approaches. Ren starts to despair at losing everything around him, which awakens his semblance, tranquility, the ability to mask negative emotions. In his calm state, he uses his semblance to help Nora and himself survive the destruction of the village. In present day, Nora and Ren enter a cavern and realize that the same grim from that night years ago was once again on its way to Kuro Yuri. As Cinder trains with her maiden powers, Tyrion returns to Salem and is distraught when she expresses disdain for his failure. He then lashes out at a nearby grim in anger, giving in to his sadistic urges, repeatedly stabbing it without letting it die. Ren and Nora rush to the village and find Ruby and Jon unharmed, mere moments before the monster they feared arrives. The Nukulavi. The battle is brutal as Team Ranger struggles against the beast, Jon gets Kro to safety and Ren begins to give in to his anger as he fights. Nora brings him back to his senses and through their teamwork, Ren is able to finish the Nukulavi with his father's dagger that he had kept all this time. The smoke from the Grimm's demise alerted a nearby airship and Team Ranger is picked up to be brought safely to Mistral. Things are looking up as Kro is healed and they have finally arrived at their destination, but unbeknownst to the group, Arthur Watts has already made contact with Salem's informant, the headmaster of Haven Academy, Leonardo Lionheart. Back on the island of Patch, where Ruby's journey to Mistral began, Yang had stayed behind, dealing with her own inner turmoil and the trauma of losing her arm. She is gifted a high-quality prosthetic from Atlas by General Ironwood, but doesn't feel comfortable wearing it. After her father is visited by a pair of her old professors, Yang realizes that she's not only holding herself back, but also those who care about her. The next day she puts on the arm and steps out of the house. It is time for her retraining to begin. Yang and her father begin sparring, as Tai teaches Yang to fight without relying on her semblance. He opens up to Yang about her mother, Raven, the traits that he admires about her and that it was her flaws that broke their team apart. Some time later, Yang paints her prosthetic arm and is ready to leave. Tai asks her whether she is heading to find her sister or her mother. In the Kingdom of Atlas, Weiss is sought out by her butler Klein to meet with her father. As Weiss solemnly walks through the lavish Schnee Manor, she encounters her brother Whitley. And after an unpleasant exchange, Weiss enters her father's study to find him arguing with James Ironwood. Ironwood had imposed a dust embargo to other kingdoms under the pretense that some may intend to start a war. And this has cost Jacques Schnee, current head of the Schnee Dust Company, a lot of money. After Ironwood leaves, Jacques pressures Weiss into singing at an upcoming charity event the company is holding for Beacon, the purpose of which is to promote the Schnee family image using Weiss, a survivor of Beacon, as a selling point. Weiss reluctantly agrees and as the day of the charity event arrives, she is displeased at the ignorant nature of the people in attendance. As she overhears a woman claim that Vale had it coming, Weiss loses control of her semblance momentarily and summons a grim in the middle of the crowd. As it goes to attack the woman, Ironwood shoots it down before leaving the party. Afterwards, Jacques scolds Weiss for her behavior, claiming she disgraced the Schnee name, to which Weiss responds that it's a name Jacques only married into. Jacques retaliates by slapping Weiss and promptly informs her that she has been disowned, that Whitley will be the new heir to the company. After confronting a smug Whitley and crying in her bed, Weiss pushes herself forward. While working on her summoning, Whitley interrupts to flaunt that he is free to go wherever he pleases, while Weiss is confined to her room. After dismissing Whitley, Weiss manages to successfully summon her knight, the Arma Gigas. And as Klein comes to see if she's okay, Weiss asks him for a favor. She has decided to escape the Schnee manor, and as she leaves, Weiss overhears Ironwood and Jacques arguing again, this time about Ironwood's decision to close Atlas' borders completely. Weiss then leaves through a secret passage in the library, saying a fond farewell to Klein and boards an airship headed for Mistral, hoping to meet her sister there. After the fall of Beacon, Blake had run from her team, and had boarded a ship leaving the kingdom of Veil. While aboard the ship, she was attacked by a seafay long, a monstrous sea dragon Grimm, and with the help of a stowaway, she and the crew of the ship are able to defeat it. Sun had followed Blake, thinking that she was aiming to take on the White Fang alone, but that wasn't what she was doing at all. Blake was heading home to sort out her own turmoil, home to Menagerie, and now Sun was along for the ride. Outside of the four kingdoms, there is another continent that is populated, the island of Menagerie, gifted to the Faunus after the Great War as a symbol of equality, but this island, largely covered in desert, was almost entirely uninhabitable, with only one major settlement, the city of Kuo Kuanah. The Faunus make do with what they have, in hopes that one day true equality can be achieved, and it is this inequality that sparked the creation of the White Fang. After arriving at Menagerie, Blake heads to her home, and we meet her parents, Kali and Ghira Belladonna. As Sun makes his awkward introductions, members of the White Fang, Corsic and Finnic Albain, pay a visit to Ghira, who was the previous leader of the organization. Blake and Sun quickly tell Ghira about the White Fang's involvement in Beacon's Fall, and the Albains blame it on a splinter group led by Adam Toros. Ghira sends them away, and as they leave, they decide that they should inform their brother Adam about Blake's arrival in Menagerie. Blake is then able to reconcile with her father, after she had chastised him for leaving the White Fang when she continued fighting. As Ghira is comforting Blake, Sun interrupts them. He tries to tell her that masked members of the White Fang are in Menagerie, and as Blake is refusing to listen, they notice a masked spy watching them from the darkness. As they give chase, the spy is revealed to be Ilya Amitola, a chameleon Faunus from Blake's past in the White Fang. Blake is able to take Ilya's scroll, but she injures Sun and flees, warning Blake that she should not have come back. Sun is greeted by Blake as he wakes up, and she tells him that this was the reason she ran. She cared about her friends and didn't want them to be hurt because of her. Sun then points out that she's not being selfless, that shutting out her friends hurts more than any physical injury. Ghira then enters, revealing what he had learned from Ilya's scroll, that Adam plans to overthrow the leader of the White Fang, Siena Khan, and repeat the fall of Beacon at Haven Academy. They try to convince the Albane brothers of this plan but are unsuccessful, and shortly afterwards, Adam follows through with this plan, murdering Siena Khan and taking control of the White Fang organization. Ghira holds a press conference to try to convince the people of Menagerie that Adam's way is wrong, urging them all to join him in going to Haven Academy and protecting it from Adam's attack. The people are wary as Ilya interrupts, calling the Belladonna's traitors before fleeing the scene. Afterwards, Blake and Sun try to gather signatures of those willing to go and help defend Haven, but are entirely unsuccessful. Blake talks to Sun about Adam and Ilya, where she says she wants to help Ilya the way that Sun helped her. At the same time, Ilya is summoned by Corsac and Finnic, who tell Ilya about Siena's death, and that they must now silence the Belladonna's. Blake, however, shall be captured and brought to Adam. Blake then receives a note from Ilya, wanting to speak alone, which turns out to be an ambush. Ilya reveals that she had romantic feelings for Blake in the past, but Blake was too busy with Adam to notice. Now Ilya planned to kill Blake's parents and send Blake to Adam. But Blake had anticipated the meeting might turn out this way, and after Ilya leaves, Sun steps in to rescue her. Both of them rush to Blake's home as it is currently under siege by the White Fang. They arrive as Gira fights the Albane brothers alone. He tells Blake to go and help her mother, and as she does, Blake encounters Ilya. The two begin to fight as the mansion catches fire. Ilya becomes conflicted between helping Blake or the White Fang, as Gira and Sun's fight with the Albanes intertwine with their own battle. During the fight, Gira and Sun save Ilya from being crushed, only for Finnick to die while trying to kill Gira. The rest of the White Fang is subdued, and Ilya breaks down, regretting what she has done. The Belladonna's burning home draws a crowd, and Blake addresses them, causing many Faunus, including Ilya, to rally in support of their cause to defend Haven from Adam's attack in two weeks' time. Back at Haven Academy, Crow takes Team Ranger through the city of Mistral to meet with Leonardo at the Academy. Seeing the Academy deserted, Crow berates Ilya for not keeping the relic protected. It is revealed that each relic is sealed within a vault that only a specific maiden can open, so Ilya gave leave to all of his staff after fighting Grimm on the Kingdom's borders. Crow reveals that he thinks the lost spring maiden is with Raven, and Ilya delays pursuing her, later telling Salem of the new information so her forces can get to the maiden first. Afterwards, Oscar finds Crow at a bar, and after drinking his fill, Crow brings him back to Team Ranger, where he reveals to the group that he is also Professor Ozpin. You have silver eyes. Oscar demonstrates that he and Ozpin can switch control of Oscar's body, and Oz tells them that he was cursed by the gods for not being able to defeat Salem in the past, his soul reincarnating to a new host with every death, retaining all the memories from his past lives, as each of their souls eventually merge together. Oz is suspicious of Leonardo and agrees to train a Team Ranger and Oscar while Crow seeks out other huntsmen to help with their cause. During their training, Oscar shows quick improvement, but admits to Ruby that he is scared and asks her how she can handle everything that's going on. Ruby admits her fears and tells Oscar about the loss of her friends, Penny and Pira, that they would keep moving forward if Ruby had perished, and that she will do the same. Ozpin then remarks to Oscar that Ruby has a spark that inspires everyone around her. Meanwhile, Weiss continues her flight to Mistral when her pilot gets a distress call. They try to avoid the conflict, but the nearby Grimm take notice. Lancers, nasty wasp-like Grimm that prey upon airships in the area. Weiss fights from the cargo hold, using her glyphs to keep herself in place. Using her dust attacks and some quick maneuvers from the pilot, they are able to take out the small Lancers, leaving only the queen remaining. Weiss summons her knight to her side and is able to defeat it, but the ship takes too much damage, falling to the ground on the shores of Mistral. Thanks to her glyphs, Weiss survives the crash, only to be greeted by the Bronwyn tribe, and their leader Raven, who takes Weiss prisoner in hopes of ransoming her back to Atlas. While imprisoned, she meets Raven's Lieutenant, Vernal, who tells Weiss that her sister went back to Atlas when Ironwood closed the kingdom's borders. No help was coming for her, and so Weiss takes matters into her own hands, summoning a smaller version of her knight to break her out of the cage. Not long before this, Yang had taken a ship to Mistral, along with her bike and ultimately had decided to seek out her mother. Stopping at a rest stop, she encounters a member of her mother's tribe as he tries to hit on her. After knocking him around a bit, he agrees to take Yang to the tribe's camp and tries to ambush her once they get there. Yang easily deals with them and marches into the camp to confront her mother. Raven congratulates Yang on finding her, but Yang only came to Raven to use her semblance to create a portal to Crow, and thus to her sister Ruby. With the ensuing commotion, Weiss takes the chance to break out of her cage and is reunited with Yang. Before a larger fight can break out, Raven orders Vernal to summon Lightning and warns that more conflict will draw Grimm. Raven then invites Yang and Weiss inside to tell them the truth about the fight they are entering. She reveals to them the truth about Salem and Ospin's war against her, that Ospin had enlisted her team in the past to join the fight, opened their eyes to the terrifying truth of the world, and gifted Raven and Crow the ability to turn into birds. She then opens a portal for Yang, giving her the choice to stay and have their questions answered, or to go to Crow and join Ospin's hopeless war against Salem. Much to Raven's dismay, Yang chose to go, only wanting to see her sister. Meanwhile, Crow had been scouring the city of Mistral for any huntsman to help recover the Spring Maiden and defend the relic, but none of the huntsman he knows or any other huntsman for that matter were available in Mistral at all, most having not been seen for weeks. As he wondered what could have happened to all of them, a portal opens behind him, and soon after, the three members of Team Ruby are reunited after months apart. The members of Team Ruby and Team Juniper share stories of their adventure and reflect on how far they have come since the last time they saw one another. Later, Yang and Weiss are filled in about the maidens, magic, the relics, and that Ospin had gifted a small portion of his magic to Crow and Raven. It is revealed that Oz can still use magic now, but it is dwindling after giving large portions of it to the first maidens centuries ago. He gives them all a choice, to stay and fight or leave, and all choose to stay, with Yang demanding that he stops keeping secrets from them. The next morning, Ruby, Weiss, and Yang enjoy their time together, and Ruby wishes that Blake could be there with them. Yang, however, becomes angry at the mention of her name. Later, Weiss talks with Yang, both confiding in one another, and Weiss tells Yang that Blake was dealing with her own burdens while at Beacon. Weiss reveals that she sees her teammates more as family than her own, and that Blake likely feels the same. She will be there for Blake when she returns, and is there for Yang whenever she needs. Ruby then talks with Ospin and Crow, where it is revealed that the relic under Beacon Academy remains safe, that Ospin took extra precautions with that relic compared to the vaults under the other academies. Meanwhile, Cinder, Watts, Emerald, and Mercury arrive at Raven's camp. They confront Raven, wanting to bring Vernal with them to open the vault under Haven Academy. After hearing of the White Fang's planned attack on the school, and Lionheart's loyalty to Salem, Raven accepts on the condition that Crow be killed. All the while planning to take the relic for herself in the confusion of the battle. Cinder agrees, and alters their plans to coincide with the White Fang's attack, hoping for a chance at Ruby as well, much to Watts' dismay. Two days later, Crow is called to meet with Lionheart, and everyone goes along with him. The meeting turns out to be an ambush, as Raven is waiting, quickly opening a portal to allow Cinder, Emerald, Mercury, and Vernal to arrive. Hazel then enters the room as Adam and the White Fang begin setting explosives around the school. The battle for Haven begins as Jean attacks Cinder. Crow fights against Raven, Emerald confronts Ruby, Yang gets a brief rematch with Mercury, Nora and Ren take on Hazel, and Oscar sneaks up to attack Lionheart. Leonardo sees Oscar using Ospin's weapon, and realizing that Ospin had reincarnated, decides to give him over to Salem. During the battle, Weiss is bested by Vernal, who breaks Weiss' aura, as Jean charges Cinder. Ruby sees Jean's actions, and her memories of Pira's final moments cause her silver eyes to react. Emerald quickly knocks Ruby out, but the silver eyes left Cinder weakened, allowing Jean to attack, narrowly missing Cinder's head and chipping her mask. Infuriated, Cinder conjures a spear and impales Weiss, only to watch Jean fail to protect someone yet again. As Jean rushes to Weiss, Oscar gets the better of Lionheart, who then tells Hazel about Ospin's return. This enrages Hazel as he begins to inject dust into his body and attacks Oscar.  Jean's desperation to save Weiss unlocks his semblance, allowing him to use his own aura to amplify hers, healing her in the process. In the confusion, Cinder, Raven and Vernal enter the cavern under the school, containing the relic of knowledge. As Vernal approaches the vault, Cinder freezes Raven in place, and impales Vernal with her left arm that had been replaced with Grim. Cinder is confused, finding no magic within Vernal, and as Vernal collapses, Raven reveals that she is the true Spring Maiden. Jean continues to heal Weiss as Ren is defeated by Hazel, and Nora rejoins the fight. Thanks to her semblance, Nora overpowers Hazel, launching him out of the building to where Adam and the White Fang are setting their explosives. Blake then arrives with the Faunus of Menagerie and the Mistral Police Force in tow. Adam attempts to detonate the bombs, disregarding the safety of his allies, but Illya had already disarmed them. He orders the White Fang to kill everyone present, and the battle between the Faunus groups commences. Hazel is then dragged back inside as Weiss is recovered and has summoned a Queen Lancer to her side. Yang gets past Mercury and heads to the vault as Raven and Cinder battle one another, the cavern collapsing around them due to their immense power. As Cinder and Raven reach their limits, in a final act, Vernal attacks Cinder, distracting her and allowing Raven to send her plummeting off the cliff, freezing her in the process. As Raven turns back, Vernal had died. Raven thanks her one last time and proceeds to open the vault containing the relic. Before Raven can enter, however, Yang comes to confront her. Through their conversation, it is revealed that Raven killed the previous Spring Maiden and that she is afraid of Salem. Yang resolves that she will take the relic instead, so Salem will come after her and she will be waiting. After shedding a tear, Raven departs and Yang claims the relic of knowledge, breaking down after doing so. Above ground, the White Fang forces have surrendered and Adam flees the battle knowing that he has lost. After Hazel manages to destroy Weiss' summon, Blake joins her team in the fight against him, Emerald and Mercury. Seeing the battle no longer in his favor, Lionheart flees and Salem has a seer grimm kill him for his cowardice. Yang then returns with the relic in hand, prompting Emerald to despair at the realization that Cinder has fallen.  Hazel and Mercury make their escape, carrying the unconscious Emerald with them as Adam watches from a distance. The battle for Haven was won. The White Fang will no longer follow Adam, and Ghira is resolved to start a new Faunus Brotherhood, striving for a more promising future. Team Ruby is reunited and Oscar passes a message to Crow from Ozpin. They must bring the relic to Atlas. And this brings us to the end of Oymes 4 and 5 of the Ruby series. Haven has been saved but is now left without its defenses. With the relic in hand, Team Ruby sets their sights on Atlas, the Kingdom to the North. Let me know your favorite moments from Vol. 4 and 5 in the comments down below, and I'll see you guys in the next episode of Ruby Recap.


Gary Stein along with the head coach of Stevenson football Ed Hoddle a lot to unpack here in the postgame coach big win over FDU Florum 20 to 7 the final Let's just talk about the defense in general. This was a high-powered offense coming in you guys did a really good job of shutting them down Yeah, I mean they're much much improved You know even even from last year. I thought they were pretty dynamic last year You know, they've got a running back now that you know He's tough and you know, they got a fifth-year senior receiver and you know sophomore quarterback that I mean He runs for 100 yards game and and you know, so coming in, you know, we spent a lot of time, you know, coach Hollings coach Jones myself you know building a game plan to kind of Limit some of those opportunities and you know, the guys went out and executed it at a very very high level today I mean, I could name names here and we can stand here for the next 30 minutes and do that So I don't necessarily want to do that But just collectively I thought you guys got good pressure up the middle But you also held the edges it looked like and I know one of your sayings is just just do your job It just seemed like everybody was stayed in their lanes and did their job today Well, and we talked about that a great deal You know do do your job and you know, the plays will come to us and you know again hats off, you know to I think we played Seven or eight different defensive linemen today We had linebackers coming down putting their hand the dirt playing some some rush and stuff and our SWAT package and You know, but again, right? Like you can talk about that D line group all you want, but the secondary I mean those guys were pretty lights out all day Speaking of plays yards after the catch is a is a stat that doesn't get looked at a lot Sometimes maybe by the coaching staff it does but today three big plays all big yards after the catch Two of them on offense led to red zone touchdowns and the last one was booze for the touchdown Just talk about your ability to break tackles and be playing space today offensively Well, I think what you saw today is you know a result when you when you're able to break tackles You know, you you don't have to look real far you look in the weight room and you know booze has been a guy that's that's you know the weight room guy and You know the ability to do that. It isn't necessarily a skill or an ability But it's a it's a strength issue and you know, I love to see those guys breaking tackles and and you know Getting the yards after catch. It's awesome to watch So E. J. Marquez apparently now is the number one back not not a lot of yards on the ground today Just like last week against like homing but kept the drives alive kept kept the clock running, etc And he seems to be getting into that groove. Yeah, I mean their D-line is phenomenal Number six is you know, he's an all-american It seems like every week we're seeing another all-american defensive lineman. I mean he's incredibly disruptive and hats off to him You know, I actually talked to him after the game and asked him if he was coming back And he said no, he was done. I was like, thanks man. Appreciate that Yeah, but I think it's still gonna be you know playing the hot hand, you know at that running back spot and Watching the young guys continue to grow and you know still relying on you know The two upperclassmen it's gonna be key for us the next two weeks So October football is now what we saw one more game in October, but it's turning into November football It's that time of the year now where the playoff when our playoffs, but the you know, postseason, etc You guys are in pretty good position right now. What's the message the next two weeks? What are you looking to do? Well, you know Monday, it'll be about Monday, you know, it's about getting better You know, it's a one day at a time kind of approach You know, we've got I think what you saw today in some respects was a tired football team You know, we've played eight in a row We didn't have a bi-week and look at you know, it's probably crying in my beer a little bit You know, it'd be nice to have a bi-week, but we haven't had one and you know We've been on the road the last two weeks long overnight trips got another long overnight trips play another tough football team on Saturday So, you know, we've got to manage that a little bit, but the focus is you know It's one day at a time One practice at a time one meeting at a time and focusing on being the best football team We can be every week final score again is Stevenson 20 FDU Florm 7 He's Ed Hoddle head coach of Stevenson football right here on go mustang sports dot TV


good to go condition one all I got to do is palm up grab that cocking handle pull it back to the rear hard and I'm gonna put it on what? Same!


Thank you. Welcome you all very astonished. That's so many people over here My talk is as introduced better bad self-efface for satellite data processing. How has this talk been generated almost exactly one year ago. I met Gregory and Sasha at the meet-up session here at this dev conf and They said they always like user stories and since we had one to provide we said, okay, let's do it. Let's do a Let's talk about what we use self-efface for since everybody always is talking about safe and doing the rather spark device thing and Most of them now also do the object gateway stuff But the surface at least how we feel felt that's not really much in Public so short words to me. I'm my name is Martin Spiegel. I work for Jaffa. That's a rebrand of catalysts game Behar individual software development in North Korea and yeah, I'm master of the classroom And I'm also not afraid of touching pay a P PHP sometimes I have basically three chapters for you including final words and I tried to find the sense of race from a song a very much like Imagine from John Maddon's old courtesy for that with him imagine doesn't happen the key word here is heaven And there is actually having a bath was and a lot of companies actually do something there So we have a lot of satellites there We have that's actually a picture of a satellite called sent in a 1a It's part of the Copernicus program of the ESA European Space Agency Which has some instruments and board in that case specifically their radar instruments so they're taking radar pictures and They produce a lot of data also airplanes which fly around and have here an RGB camera on it and here a laser scanner They basically Do something that's then called photogrammetry. So you take two pictures of the same area Roughly overlapping a little bit and then you can calculate out of that if you know the exact height and the Different angle you have you can do three D point Mesh walk clouds. We will see later something of that So and what did we start to do this data processing? We met the academic research group in France that Doing a cat algorithm that was called grass that's that generalized material of our soul and Surface properties. So basically what do we have in the air and that algorithm was apparently very great They did find much more to get much more out of the satellite data than everybody else did but they had one problem They were academics. So that's not a certain problem but academics tend to do at their great in Developing algorithms, but they're not so great In developing cold that's fast. So they had a quite good code base which worked but it took them very very long to process such files and then we came into play and we Improved the speed of the algorithm at the first stage about the factor of 100 so and Somehow that joint venture came started and then we met other companies which had basically the same problem They had a good idea. They had good techniques. They had good algorithms But they didn't have to know how and didn't want to build it up. That also came up with the time the knowledge to do this in a large scale so and Yeah, what then happened? We did some processings for USA and omets out and other stuff and You know to get you an idea what data do we process is here. These are concrete pictures. So the upper one is from Sentinel-2 image. That's actually an image of criminal Sentinel-2 has optical sensors in it so you can do basically for the graves and you can then use that Here we have a 10 meter resolution at the ground level To calculate something out of it. So you can try to detect particles in the atmosphere which fry over Brennan the other thing is here we have so that's the base material and Give you a roughly an idea of what we're talking about if it would take Every day of the satellite sentinel-2a mission That would be roughly of every instrument that satellite has on it. We would have about roughly seven terabyte per day So and what we need to do is so that we can process all that stuff We need to massively parallelize that so we built up also a compute cluster with a lot of compute virtual machines and they do the Concrete processing in parallel, but that also needs Some kind of storage which they can access in parallel so they can share that in between products Normally generate so normally it's something like we do a first round on some picture sequences Then there are some mid-term products We then shared that amongst the others because the other processes need that information to further iterate and gain speed and so we Apparently had something where we need shared storage The next thing is most of these algorithms even those we own basically so we have the We can influence the code They're not very object storage oriented So that's not that easy to Rewrite them to use that and there are also a lot of algorithms where we don't even have the source code for us So we can't even say yeah, well, okay Physis them accesses bullshit. Let's do it in object oriented way Which would be most of the times better, but we can't do it and we can't use it there So this wouldn't make sense to go that path So and here I tried to list from a s5 p-satellite So there's a sentinel 5b and a sentinel 3a to give you an idea What the difference is between what we have here we have that's for exactly one day in July 2019 this satellite produced 14 files exactly 14. We will see could you can see here They are all Basically roughly the same size all over the day and then we had another set the light the s sentinel 3a Also, just a random day in that case in the year 2018. We have 150,000 files and we have 150,000 files not only large amount, but also range from 40k to 30 megabyte something like that Why is that important because normally you tend to do optimization also on a Block size level, so you want to say okay, it would be optimal if all of my files. I don't know 64k then I could say and trim my strip size in the south of s to 64k and then that would be quite good That's not the case. That's not easily possible here so not only that we have Really a huge difference in the products they we have to consume, but they are also in The products itself quite different Before we go any further. I would like to show you some products. What what we make out of that. So that's actually This is a grasp Processing which shows three points in that's where is it the Cape of Rome Milo's and soccer What you can see here is the darker the blue the less the air is polluted and If it goes to red or yellow then air is very polluted. That's concrete for project where they plan to do Certain research Not a research Certain locations where people with a lung obstructions can Have a medical therapy and there it's very important to not have very polluted air So that's something very concrete or here we have the smoke plumes in the atmosphere and South America during the fires in The Amazon so you can see it quite good here. You have a lot of dust and pollution This is a thing the digital twin city of Frankfurt. That's something that's generated out of photochromatic Stuff, that's the thing where I said to overlapping pictures and then you can create Point clouds out of that. So here that's Still from the picture and here you can already Recognize that's the generated point out of that. We did that as you can see here for Frankfurt We did that even for Munich You can use that for what can that be used you can use that for development of Country areas or city areas you can do it for concrete calculations You can now really measure by heart that a point from here to that roof something like that forever. What do you need? So Another points in the challenge as I said before already Most of the time we need to process the data every day since it's such a huge amount and we wouldn't come up or To say with with that if you wouldn't do the processings every day We are not always in control of the algorithm source So we can't easily say okay, let's keep the file system part and use something else so that also doesn't work out for us Since I said we almost get seven or roughly about seven temporary per day into our storage We need something that's K level up to a very large level because Project did some time point in time said okay. I always need the last ten years of A lot of satellites and a lot of satellites mean a lot of data It's all because written so that's another thing We didn't want to buy I don't want to name some concrete vendors, but large boxes where we have pictures afterwards We shall be lean and Asia so it should be easy. So this was the main idea To change things So don't have buy a product where you can just do things just in one way and in the water so That immediately led us to it's easy if you try it Key word here is trying We didn't say well Let some vendors get us some sales representatives and then we just buy what they What what has the best price we said, okay? We want to evaluate it and what did we evaluate first of all is of course we had Think about classic or new age. What do I mean by that classic is for me the direct storage you buy from some Wenders on large and buy one two three four of them and live with the shortcomings they have The new age would be soft to define storage with all It's inclinations and we from from the beginning said okay We want something that is at least able to do block and file storage object store We didn't care so much about Just has to write an excess path in terms of blog and file storage low cost and any hardware is also fine because we didn't want to get a Vendor login we didn't want to get into the situation where at some point in time We suddenly recognize oh my fuck we can't own we can't buy anything else than HP or something like that We have we wanted the flexibility to access it in multiple ways and of course the scalability Was also very important fact What we took a look on so what we evaluated on the paper that really was on paper just reading through the dogs and trying to See if that fits first where last year fs. It were IBM spectrum care. That was actually gbfs code back then There were a cluster and okay. There was so that were the ones that Came up for us Then we had the discussion do you want something like close source that would have been gbfs gbfs better great Basically does it shop quite good as far as I'm told I mean it's expensive Yeah, you just can scale if you'll buy more licenses and these licenses aren't quite cheap But at least it does it shop. Why good, but then we said okay we everywhere else we build an open source So why should we start doing it here in our way? So that immediately led to the extinction of of gbfs Then we said okay, we saw there's exactly one product That was the main difference of self between all others at least for us which says or claims But the integrity is an vital part of the solution It's not that we say okay. We do the the distribution thing and stuff and that would have been last row also Okay, so but we and the for about integrity at the end of the day the rate controller is responsible or something like that That was the only solution that did that we needed posing compliance and the file system level Yeah, and that led to the fact that we wanted to use self what we did then as we said, okay Let's establish a plan. We at least let's try one more one other file system What I meant is then forget about plans life doesn't like them I Don't know exactly the time frame anymore after the roughly about two months of testing So we built up a test set up where we could Throw away everything in a few minutes and build up a new cluster. So that was what was done for safe and for cluster But at the end of the day to be honest, we just tested said It was okay, it worked out now we had some colleagues Which already did a cluster affairs before and we consulted there and then they said okay well for that What do you need go with that? So This is just Gregory thankfully already explained the basics of set for itself. So I don't have to explain that anymore This is roughly how does set a fast work? So we have a data pool data pool is a lot of people unit in the self construct, which has a crush rule behind it Which determines where objects should be placed? We have a major data pool and we have here the client and we have the things that's called Meet the data service. These are actually the ones which know where files Live so basically that's the idea To put it quite frankly This was just needed to tell you something else then we had the evaluation of hardware So what can you do there? You can read white papers. There are actually a lot of winners right now out there Which already have something like Yeah, that's the perfect self-solution We have we can sell you a whole safe rack for that and stuff like that that has Network suppliers like Melanox or so they provide you with a lot of white paper So that's our basic idea of an adult self cluster stuff like that I mean you can read through all that you can check the prices you have to find a way of trade-off So there's basically something where we said okay, we need a lot of storage. So this is one of the main ideas It's not that easy to create a safe cluster. Not Doesn't mind if it's just a block storage or something else, which is the Multiculti animal which can do anything so you have to find a trade-off between Performance between high IOPS between a large storage amount This is something you have to decide basically for yourself and at the end of the day What proved out first or the only thing to really work out is try it Buy or talk to your vendor if you can give you a pre-sale or a demo object and then try it. That's it These pictures you are here to show you what we have right now in the class is actually HP and super micro devices We tried both of them Yeah, as said before we don't want to get an event along in the work. That's basically it's it's not like that I would have say yeah, the one is better than the other don't care really much What we did actually as I said before we did a lot of load and performance test That's that's really something you should do and the most important thing is you should identify your workload So how will your access pattern be? That's crucial Because otherwise you might run into the situation where you say hey I have a perfect cluster and it's quite good spices to have a distributed one and your users say yeah Well, that's fine and we have a lot of storage but actually the performance. Yeah, it's not that good So what we did is we created an automated sweet Roughly can look something like that so fat is a Python thing which can Python SSH scheduler he can schedule jobs for you and different masterings executed over SSH And there we had as you can see it is just for one scenario a lot of fire jobs so fire is a utility which just can measure in a large scale and with a lot of threads and stuff like that multiple scenarios for reading writing files So that's what we did what we said, okay? That's what we want to use and what we also didn't just test it from one note We said well at the end of the day there will be 20 30 40 Compute notes which will access this file system in parallel, so we have to test it in parallel So we had to build up all that stuff and Yeah, and then run the test suite And the outcome of such test suite is then exactly such a graph where we can see yeah Back then to that point in time where from where this graph is is we had the class to build up with form storage nodes because we needed the other ones for the virtual machines to provide the Compute VMs And as you can see it's quite impressive already with a certain number of threads accumulate that you get around roughly 3500 megabyte per second Which is good because it was a 10 gigabyte internet network connection between them So no help below us How I wish that sentence would have been true a lot of times What were our challenges when we got that stuff into production first of all I have to say we Immediately switched basically with the final testing cluster into production. So we didn't rebuild it Not a real rebuild which I said, okay, so it's working fine. The settings are good. Let's get that stuff running Actually It would have been wise we would have done that from scratch But well it was like that because we already had some 100 terabyte on data on it We had one thing when we started all the trial phase We were in a data center where certain environment circumstances were not optimal. Let's say They had problems with the power supplies. They had problems with the air condition So that actually led to a lot of outage of our notes And the outage actually happened during a lot of ride and read scenarios. So and Now a thing to say we used we didn't really use redhead self storage We used the upstream for a project and we used it in a way back then when even their surface was not marked as Production ready. So it's not to blame surface for that. That's something happened back then But it just happened We didn't actually notice it until some point in time some colleagues said, hey, well, I did I did start a processing, but it didn't work out. There were some errors. Can you take a look at and then we suddenly noticed? Yeah, actually we have some loose leaves in our file system tree where we have a director in fragmentation Yeah, about half of the facts first of all They didn't care all too much because these we are as you can see perhaps on the director name something 10 files We didn't need them. So that was basically not that problem But we had somehow to deal with it because otherwise they got problems the colleagues which did the processing and It's roughly the half of that we could create could Fix with a ball with with safe provided utilities and the other half We actually really had to to kick manually some objects So we had to identify which object means that in middle data pool and had to remove it Of course during the middle data, so we'll stop and then start it again One thing then we also had to notice. Yeah, you should choose some settings wisely up front The layer tries a layer free size and range You will like to use for the front and network optimally in an optimal world You have a power front and then back in the network in a self-cluster The front of network is that that is facing to the clients. So also for surface. That's also true And we started with way too less. So we had to afterwards blow it up to a slash 18 we have right now that works until now Actually started with a slash 24, which after I don't know exactly more And more think about after one month we had too much lost for that and then we then Somebody had a great idea. Yeah, well, that's then that's the rocket Yeah, this doesn't work out in terms of performance very bad. So That's not the best thing Pull assignments to directories. So if you're like us and say, okay, you have different Important products. So have to in between products as said before you have to end product Which are very valuable because you Invested I don't know three months four months of processing time into that and you won't do that easily again Then they are very well. So we created some different pools where we said, okay here We have a higher application factor here. We can work with erasure coded things Different importance levels and we assigned these pools to different directories So there's something you can do in self-affirmation. You can say, okay this director or even this file should be in that pool That's quite cool. I think and then you mean can even do more you can not only can say I want the pool You also can define the stripe unit the stripe count the object size The thing only is if you do that and that Is used in a recursively ways or the sub sub sub director of this Directory would also have to set in pools FFS data as Long as I don't define somewhere in between the path at different setting for that The only thing is if you change it afterwards it Doesn't have an immediate impact. So if I would change for that directory Okay, now go to a different pool that would not happen by magic that in the background Somehow self manages to move all the objects from pool a to po b It's only applied for new files in there So if you want to do that, you would have to copy that file You set it on the directory first copy that file remove the old file that would work out Then you would have it in the file in the new pool Which is not quite easy if you have a few millions or billions of files So choose that up front you definitely want to have some monitoring So we have basically everything in there. We started with in scope. That was at the time of in final is something like that In scope is a project or was a project. I think they are not active anymore which had something like a quiet mix of just dashboards and Graphics and a little bit of control of the cluster also We don't use that anymore Then we had me established a home room for Mita's Grafana stack, which is still in use today Then there exists also the solution provided stacks since not the loss. I would say it's also usable Before that it wasn't it so this screenshot you can see is from the solution provided sex or from self itself That is is our main dashboard of Grafana for that thing And what we also do did is we integrated that into our chat thing. So we have everything rules in the parameters which Tell us if something is going wrong. So I just made a sequence here It says yeah cat is always D is dangerously full Or we also have something like the this compute node or data that storage node uses too much memory stuff like that That's quite fine because it directly get gets into a chat Channel where all of the team members can react to that your issue coding Yeah, the gross net ratio versus performance thing We definitely have some product trees or five system trees or directory trees in there Which we want to keep for a longer time, but we don't want to spend all too much in terms of Grow gross ratio gross net ratio on it So normally we would say okay for that importance of that we would do at least the Replication factor of three or four Some sometimes we talked about five, but then said to my project the other actually well You know what it means that means we have to buy five times the amount of storage You really want you really can use I said, okay, that's that's not that good. So we have to use a ratio coding Yeah Back then when we started with all of that it was not doable to use directly and a ratio coded pool for it some Saffa fast directory what we had to do back then was We had to add a cash tire Which is in in terms of again a replicated pool and that again have immediate impact on the performance of all of that since luminous I Think yeah, I think it was luminous. You can directly use a ratio coded pools But only if you're on blue store That's also important then you can use and we do that right now Then you can use a ratio coded pools directly for some Saffa fast directories Another thing we didn't start with we had to learn it also the hard way was you will like flash storage And you will love it At least for the meta data pools and the journaling if the journaling if you're on five stop But even if you're using blue store, you will love it to give you a roughly an idea They show measurements of a concrete Processing run when we had no flash storage for the meta data pool and we had That was the commit latency in average of 16 milliseconds That's quite high When we established the flash storage we drop that to 70 microseconds. That's a fact of 228. It's really impressive and you immediately know that is it it feels way more fluid if you use it under the Common line and stuff like that. That's really great Yeah This was something I was talking about performance influencing parts and we have something Well, it's a usage pattern in that whole cluster we call making a Holter Holter is the name of a colleague Which actually had the gift to bring the the cluster to its edge, so Inside a joke and we had to react on that and what we did was simply to separate Performance influencing parts away from from other stuff. That's important. So we created different crush rules Which make a separation between block devices and surface We also will have different crush rules of different pools with different crush rules for whole directories of the users on on certain Virtual machines whereas the satellite data and that's what you can see here basically so we have the surface data pool That's crush rule zero which should know that's not that's the default rule still Then we have said if it's made a data that's crush rule three. It's that one here where we use a replicated pool based on enemy storage and This notation here is a class notation which tells the crush algorithm to use only devices of that class and then we have again here for the Rattles block devices for shift for all the shift instance where we use it, but also for the center volume since we are also using open stack Crush rule four. That's that one here where we say, okay We separated some heart is some always these just for that usage so that the poor performance impact or When somebody does a Holter on the file system doesn't impact the virtual machines I know is that Maintenance, yeah You have to deal with version updates and upgrades the cluster size changes. We started with one petabyte We're right now at three and a half petabyte Next round is coming this year. So then it will be something about four to five petabyte It's nothing you should take Easily and you should think about it at all. So if you just would do it in a plain way get the new notes Stuff it in start it up. Your cluster will immediately work and it will everything will be fine Basically, but you will notice performance impact. So we did that in a steered way so that we can control During night relate we use the cluster balance also for that the up-map feature so that during nights We're not a lot of people are working there except the processings that that are running We can do the rebalancing and stuff like that another thing We had to decide a little bit is to rebalancing per placement groups or by by usage the volume usage it would be ideal if We could use by placement group. We can't actually use it since we have a Quite large gift then of the usage on this on certain OSDs Yeah, then we had some out there chess I told you before about the The things with the power supplies and with the cooling systems But beneath that we also had a semi broken network cable that was really nasty Since it actually was working a little bit But when you put load on it, it suddenly started to generate media errors But not in a way that you would have seen it on the TCP a P stack You just could see it in some certain proc subsystem assist subsystem of the current where it says the Manamox OVD thingy starts to count up the impact on the performance was was was really large We had some active MDS and crashes again not fall of the product itself This was during a time when it still wasn't marked a stable and yeah We had other problems with the hardware to disk sprake mine not like that But every you have to be aware and rotating this Will they are used quite heavily and at some point in time they will tell you now I've done I'm at the end of my lifetime. You have to replace me General half of it so our advice is if you start to do something like that and that range and that scale Have spa spare parts at hand It might be that in two or three years when your clusters is running And at least if you're like us and you didn't buy the super expensive window stuff and had Not a support contract with ten years with 24 seven stuff like that. Then you should put some spare parts Beside and lay that stock Yeah, what still do we have here. Oh, that's a picture for friends like it Blue store rooms It's another thing that's that's quite important. You want to use blue store. There's no reason at least we believe so to use file store anymore Sadly these graphs here.  Oh, it's not 100 But very high So you definitely want to use that use cases I Can't really tell you how much time I spent talking with all the guys who will use that cluster in a specific set of s In the next time and how they told me yeah, we need that we will have that workload We will have that access pattern and we will need that was the best thing. We will just need that much in storage forget it, I Don't know if they do it on purpose or not, but it's basically a lie to you a few days ago we had an appointment in an academic environment where the IT leader guy there was telling us Give an academic guy some storage. He will fill it up in no more than two days and second thing is you will not be Satisfied with the performance he gets so that's the bottom line you can take with you Always think a little bit Even for them and another thing is of course also the use cases evolve over time. We had a Since one and a half years something like that We're working with a company from Germany, which has also acquired a good algorithm doing in this photogrammetric stuff But as I said before already they have some really weird access pattern for example, they have some What are four hundred bytes in size control files Which are dynamically created only at the beginning of a processing run during the processing run these days The same but what they do is since they we have performance their problems They under analyze them and the song well actually They are opening and reading these four hundred bytes during a sequence of ten seconds twelve thousand times well This is not really good on the on the file system of that type Last thing is here security implication and consignments of self-efface here I would not actually use it as a public file store for anything There is what is the thing that I know is us the most is the loose coupling of pool and pause explanations So if I have I am the user who has basically some kind of a keys So there is an access management and staff that allows me to access the pool a Then I can if I know what I'm doing read anything in there at the end of the days all the data that is in there Objects and as long as I'm allowed to read the pool which I apparently have to be otherwise I couldn't use it as a suffice is then I can read also data Which I should not be able to read in terms of pause explanations So that's that's a thing. That's not that good I'm coming a little bit to the end of both us on this guy is the current solution So what do we have right now? We have as I said before open stack It also uses the same cluster they are apparently we're using the block devices Why do we use open stack because we have quite dynamically be able to Throw away all that compute virtual machines and regenerate them This works quite well with open stack. We have some terraform receipts in place for that So to give you an idea of 272 Processing virtual machines which actually take half of the cluster resources and that's already something about 1,700 CPUs and 9 terabyte of Random access memory Are generated in five minutes? So that's quite cool since we can provide the colleagues which actually do the process things a way to say okay for everything The way let's start from scratch and you can do use some number of algorithms stuff like that Yeah, we use the ansible thing for Setting up open stack on itself then we use as well also for provisioning or setting up the Virtual machine after the terraform thing has created them You're using form and for inventory management and bootstrapping the whole note. So there's also a thing we can Get a new note doesn't matter if it's a storage note Or it's if a carpet note or something else into our cluster system in roughly about 10 minutes So that's the time it takes just for formatting the disk and setting up the operating system and stuff like that And then we have our colleagues that created a glue cold user interface the processing portal Which that's some quiet unique features I think it can integrate into other independent computing facilities So we have Companies which are part of us which do the same stuff basically as we do on different Products so they're using GPFS or and and we emerald stuff like that Whole thing is container-based Slurm is used slurm if somebody knows it is a job scheduler on the in the Unix world Largely used in an academic environment and Yeah, the thing with the different points of files already told you before That's how you can see a this Stranger to have here because we can see here one thing what all of this Portal does it's heavily really heavily uses the set file system So under the mount point slash media slash slash scratch. That's actually set of s and CFFS is because it's working that good was released also used to do the orchestration stuff in there, so That really works out. Yeah, you get nice graphics out of that Yeah, you may say that I'm a dreamer We are at my final words basically This is a picture of the cluster in the upper part So exact most exactly half that thing here is the self storage and the other half is the compute nodes Resiliency we survived really multiple outages with almost no interaction immediate manual interaction necessary So there were one through season power strips. There were the cooling system failures There were discs dying disc crashing OSD's We've had once the the part where exactly in half of the cluster since we were talking about Front of stretch clusters was kind of way because the power system failed Of course at that point some point in time. It didn't work but the cool thing about it was We didn't have to do anything after park came back I don't know the time from exactly animals. I say something like two or three hours something like that everything was green again No admin doing anything I Had a similar situation at another company where we had the classic old school rec storage solution They had to work five days until everything was good again so that's quite impressive and Both does that resilience is definitely a factor. Another thing I want to give you on the way is documentation Don't just rely on the documentation. I mean staff is a really great product The documentation does not really reflect that in my opinion But I told you that last year I mean On the other side to be honest a lot of people always claim that it's better to use open source software Because you have the source and you have the power and stuff like that and how do they use it? Do they ever read one line of code? So we have to take ourselves on our nose and say okay if you're using that when we claim it That's cool then really use it and Don't blame the dogs if there is something working different than it actually should I mean, okay, dogs should also be up to date, but yeah, and last thing is don't underestimate the complexity It took us quite a while to really tame that beast In that scale mean the fellow world thing is done in a few minutes. That's working. It's cool But in that scale you will most likely have other problems Yeah, since I like to And my talks always with some meaningful phrase and you might already have noticed I'm a little bit older. I'm from the last millennium and I also like the old stuff and I thought that a little bit a thing a phrase from from the song Imagine and the storage will work as one So that's it from my side, thank you for your questions. I hope I can give answers Sorry Okay, the question was do we have any background of the data actually to be honest no That's by by concept. This is all we've said If we would really talk about backing up free thought so what the satellite data actually is roughly is free thought one Peter bite right now It would not be restorable in a sense full amount of time So you have to trust in the system itself and choose your settings your reputation your ratio coding stuff in a way That you're safe and the failure domains and stuff like that So it's we don't use a failure domain of always deal ever right now We are still on host level and the next iteration will we will move the field domain to rack level Then it's gone So that's the this year. We will deal basically with that so Tom on a Fold question was what happens if somebody accidentally delete something Yeah, well, there is there is stuff you should not be you should really not delete since it's important since I don't know six months of Possessing are in there These stuff is put into a place automatically where it's just read only for everybody else except a few That's the way we deal with it mean We could also talk there. Yeah, we could talk about doing a classical backup system. You can use a burials or I don't know How all the others are cold? There's the company which now owns redhead which also has a product in place the storage manager Backup manager. She will she will read something. I don't know where they want With that amount of data, it would be possible since we're talking there about roughly 50 terabyte or something like that but yeah Everything else this simply won't work out What exactly so the question Okay The question was what is exactly our hardware specification and if there is a server with only and we meet drives Right now we don't have us. That's the first easy part. We don't have a server with just and we meet drives We're right now in right now actually Organizing them But they won't be just and we yeah, they will and we mean you to trust it will be so yeah Right now we are thinking about the pure flash storage part of the cluster since we have some workloads with databases and stuff like that where we Definitely can benefit from the high IOPS thing of that and the cool thing is again We still have to see if it really works out in a theory. It should work out to be let that be also part of that cluster For the other nodes. It's a set before we have basically get roughly the same to the storage nodes are 256 gigabyte of memory CPU with two CPUs with 14 real cores, so that's 28556 vc bills with around 2.7 gigahertz something like that And the one notes on the left side we saw have 24 discs normally roughly about 10 terabyte And the other nodes have 36 discs. You shouldn't put really more than 36 and that's also only for a cluster with Which is volume-based should be volume-based Yeah more than one it was but I don't know exactly the calculation for my anymore quite hot at least Yeah Yeah So the question was what has this qualified cluster fs Well We as said before one demand was to have different access paths and we really Wanted so that was that? hard demand a block storage and file storage and At that point in time we didn't trust the block storage component of cluster very much So this was the disqualification basically and the other thing was that we also said, okay, we're getting We're getting the files is there and not the files is then the swift S3 compatible layer for free So yeah We will definitely need that some point in the future now. We have open shift for very much We were very much using that we even use it use it don't think after one week of last year's death com So we hacked it together that it somehow got installed and then we will need There is a colleague out there, which already should be working on that that we have the S3 part running in next week or something like that So, yeah, it made sense So some more Fine, then thanks for my side still have a great