The season kicks off with possibly the deadliest first three minutes of any American Horror Story season. The year is 1970 and three happy campers are about to hook up, but one of them keeps hearing a jingling noise. Next, she begins to remove her sweater, but while the sweater covers her eyes, turns out her pals got a knife through their kiss. Nine people were killed here, but more on that later. Next, it's 1984 and after the synth remix of a normal American Horror Story title sequence, we meet Xavier, Montana, Ray, Chet, and Brooke, who all coordinate an entire plan to all drop everything and go be counselors at a camp that they've never even heard of. Well, all except for Brooke, who would rather not abandon every responsibility she has in order to go on a retreat with her aerobics classmates who barely know each other. After making a great decision, Brooke goes home and gets attacked by the Night Stalker, or real-life serial killer Richard Ramirez. He tells her that, although he missed many chances to kill her that night, he plans on sealing the deal on another day. By the way, Brooke did manage to do a number on the Night Stalker with a frying pan. So with a serial killer on her tail, Brooke decides to hide out with the aerobics gang alone in the woods for a few weeks. They stop for gas, and the attendant, Ed, does a great impression of the crazy guy from Friday the 13th.  Then, I know what you did last summer happens as they hit a hiker with their super cool van. The aerobics besties get to camp and meet Margaret, Rita, and Birdie. Rita, the camp nurse, fixes the hiker who nobody admits that they hit with their car. Next, they all gather around the campfire, and instead of singing their campfire song, Rita decides to tell them all about the murders from 1970, also known as the opening scene of this episode. She tells them about Mr. Jingles, also known as Benjamin Richter, who learned to kill in the war and loved it so much that he wanted to kill again. Margaret decides to show the rest of the gang her lack of an ear while correcting Rita about the body count of the murders. Turns out Margaret was a victim of the 1970 camp Redwood murders. However, her story about bubbles and Jesus is suspicious to say the least. Uh-oh, Mr. Jingles escaped from the asylum. Some detectives find a newspaper that suggests that Mr. Jingles is now after Margaret. On his way to Camp Redwood, Mr. Jingles stops at the gas station, just like our aerobics pals. But Mr. Jingles isn't here for gas, he's here for murder. Now Mr. Jingles is at Camp Redwood and he kills the hiker and once again, Brooke evades murder. She tells the gang that the crazy ear guy is here, but after not finding the hiker's body, they all decide that Brooke is just insane. Next, the pay phone rings, which is weird because we've already established that the phone lines were down. Eager to be murdered, Brooke investigates alone in the middle of the night. She discovers that nobody is on the line, except maybe some jingling keys and grunting. But then Richard Ramirez walks out of the woods because maybe Brooke really is crazy. But the episode begins with Karen warning Margaret about Mr. Jingles' escape, to which Margaret responds with grabbing her gun and scaring Karen away.  Then Brooke is the only one who hears about Ed's murder on the news, so once again, everyone thinks she's just paranoid.  Since it is night time, the boys all take showers, but Xavier heads back to the cabin for his beach towel. On the way, Margaret turns out the lights, and Xavier gets grabbed by Blake, his sugar daddy who is now threatening to release a porn that Xavier did unless Xavier agrees to make more. Xavier bargains with Blake and says he has someone that can replace him, and then takes Blake to the showers to look at Trevor. While Blake peeps, Xavier makes a run for it, and then Blake gets a spear through the eye. Meanwhile, Richard Ramirez is really also here at Camp Redwood and finds Brooke at the dock. She beats him with a paddle and manages to get away when Ramirez is distracted by, surprise, the hiker that Brooke just saw dead on the door in the last episode. Richard guts the poor guy and moves on. Until later, he finds the hiker again, and to his shock, he's still not dead. So Richard kills him again and finds his ID, which is dated 1970. Then, all of the sudden, the hiker is gone again. Richard Ramirez breaks into Margaret's cabin, and the pair have some tea and discuss Richard's troubling childhood. Then, Margaret herself runs into the hiker and is weirdly very much not at all phased by the ghost in the time loop. Then, the gang is all made aware that Blake is dead, and there really are some killers at the camp. So they try to escape in the van, but crash into Margaret's car after nurse Rita jumps in front of the van. Rita, who had been attacked by Mr. Jingles, has a wound on her chest, but is still living, and she tells the gang that she has a car, but she needs to go get her keys from the infirmary. Trevor also has a motorcycle, but needs his keys as well, so the group splits up and they all get spooked when they both have unknown threats banging at each door. Firstly, Richard Ramirez nearly kills Ray, but Chet saves him, and Ray, Chet, Brooke, and Rita are all able to escape. Then, two Mr. Jingles impersonators throw some flaming feces through the window of the cabin that Trevor, Montana, and Xavier are hiding in. Then Jingles kills his two fans, and the gang is able to escape. Chet and Ray fall into a pit of spikes, then Jingles spares a Jingles impersonator who is probably legally blind, given that he has no clue that he's not talking to a rubber mask. Next, Rita sedates Brooke after Brooke suggests leaving to get the police. Flashback to one week ago, Donna Chambers, the character formerly known as Nurse Rita, meets with Mr. Jingles at the asylum. He tells her that he doesn't remember killing the kids in 1970. Rita then tells Jingles that she needs to study him in his natural habitat, so she convinces him to escape using her plan. Another flashback where Donna stalks the real Nurse Rita and eventually overpowers her and takes on Nurse Rita's identity. Then, back at Camp Redwood, Donna drags Brooke's unconscious body off screen. Back at the pit, Ray reveals to an unconscious Chet that a fraternity pledge died under his watch during Hell Week and that when he tried disposing the body, turns out Guy wasn't dead, but Ray was unable to stop the car and the car went off a cliff, thus making Ray responsible for the death of that pledge. Three days prior to Ray leaving for Camp Redwood, Ray hears that the police have found the car that the pledge was in, thus prompting Ray to go to Camp Redwood. Then Ray climbs out of the pit, leaving Chet alone with a spear through his chest.  Then Ray finds Montana, Xavier, and Trevor, but lies to them about where he's been and doesn't mention that Chet is dying. Ray and Montana go to try and get help via Trevor's motorcycle and Xavier and Trevor find Chet who tells him that Ray left him to die.  Richard Ramirez finds Ray and Montana and Ray leaves on the motorcycle, abandoning Montana. Then karma comes in the form of Mr. Jingles decapitating Ray before he can leave the premises of Camp Redwood. Now back to Montana and Richard Ramirez who are making out, but once that's finished, Montana asks Richard Ramirez why the hell he hasn't killed Brooke yet. Episode four kicks off with a flashback to Montana leading a midnight aerobics class. Richard Ramirez stops by and kills an aerobic student who was rude to Montana. Montana is into this and asks if he is willing to kill for her again. She then reveals that the groomsman who was killed at Brooke's wedding that Brooke was accused of sleeping with is actually Montana's brother and Montana believes Brooke is responsible for his death. While tending to Chet's wounds, Trevor and Xavier realize that Blake's car is a way out of the camp. So they split up to find Margaret and Birdie. Margaret finds the two dead Mr. Jingles impersonators and Trevor tells Margaret that the real Mr. Jingles killed them. Margaret refuses to go to the car with Trevor because she has a gun and believes that she can protect herself. Xavier finds Birdie who doesn't believe Xavier about Mr. Jingles being back until Jingles steps foot into the mess hall himself. Xavier hides under the table and Birdie bonds with Jingles until Xavier makes his presence known which causes Jingles to stab Birdie repeatedly and lock Xavier in an oven set to broil. Brooke awakens in a shed after being sedated by Donna. She breaks out but then activates a booby trap causing her to be stuck in a net. Donna finds Brooke and reveals that she set the traps and she reveals to Brooke that she's behind Mr. Jingles escape and his return to Camp Redwood. Donna then leaves Brooke to be killed by Mr. Jingles. Then clinging to life, Birdie frees a severely burned Xavier from the oven. Birdie asks Xavier to finish her off and put her out of her misery which Xavier reluctantly and painfully does. Montana finds Brooke and tells her that she'll go get help and meanwhile Margaret has a flashback to the summer of 1970 where Benjamin carves her a wooden bear revealing a romantic past between the two characters. Benjamin promises that he'd do anything to protect Margaret. Back in 1984, Montana tells Richard where Brooke is so he goes there but he is then confronted by Mr. Jingles. So the night stalker and Mr. Jingles duel which ends in Richard Ramirez's head being impaled on a tree. At the same time, Richard Ramirez stand Montana confronts Mr. Jingles stand Donna which ends with Montana slashing Donna with a branch and getting away. Mr. Jingles then heads to Margaret's cabin and he tells her that he's there to finish what he started to which Margaret responds with revealing that she committed the Camp Redwood murders in 1970 and let Benjamin take the fall. Flashback to 1970 and Benjamin is beaten, electrocuted and made to believe that he is guilty and detached from reality.   Chet, Montana and Trevor hear the gunshots and Trevor goes to check on Margaret. Trevor sees that Margaret has seemingly killed Jingles and Margaret then stabs Trevor and removes his ear. Xavier stumbles upon a living Mr. Jingles who tells Xavier that it was never him who killed the campers but suddenly Brooke finds Xavier and Mr. Jingles vanishes and they then hear an explosion and find Margaret, Chet and Montana by Blake's burning car leaving them with no escape once again. Margaret tells the survivors, Chet, Brooke, Montana and Xavier that Jingles attacked her and killed Trevor.  A flashback to 1980 where Donna Chambers walks in on a corpse that was mutilated by her father. Her father continuates that he's been murdering people his entire life but Donna tells him that he was not born that way and that she can help him. Donna's father then stabs himself in the neck killing himself then back at the never ending night at Camp Redwood, Richard Ramirez says he was reborn and resurrected after being killed by Jingles. Richard also tells Donna that Satan showed him the night of her father's death and he tells her that she is no different than her father. He says that she is responsible for the 1984 Camp Redwood murders. Then Richard summons a ghost or at least a physical embodiment of Donna's father who advises Donna to follow in his footsteps. Xavier tries to start a forest fire but Margaret knocks him out before he can do that. Margaret tells Chet, Brooke and Montana that she saw two people camping across the lake and that taking a boat across would be their best bet at survival. Montana volunteers Chet as the first boat ride recipient. Then Montana tries to attack Brooke but Brooke sees Ray walking outside the window before Montana is able to attempt murder. Brooke goes to find Ray who has little memory of what he's doing and what he has done and in the middle of the lake, Margaret gives a church sermon to Chet before telling him that she never saw any campers across the lake and that that was just an excuse to get everyone alone with her. Margaret then kills Chet by chaining him to an anchor and drowning him in the lake. Brooke and Ray hide out in the mess hall and they end up hooking up and Brooke confesses that it was her first time. Xavier wakes up and Donna enters the cabin. Donna tells Xavier that she released to Jingles prompting Xavier to attack and chase Donna. Brooke finds Ray's decapitated head in the fridge which informs both Brooke and Ray that Ray is in fact a ghost. Brooke runs off and tells Montana that Ray is a ghost and Montana strikes Brooke on the head and now it's 20 minutes till sunrise and Montana reveals her villain Vendetta to Brooke. Brooke overpowers Montana and gets away. Meanwhile, Mr. Jingles tells Donna that she shouldn't have brought him here and Donna says that she was just trying to figure out if her father was always a killer. Jingles tells Donna that Margaret committed the 1970 murders and framed him. Donna tries to get Jingles to kill her but Jingles says he will only kill one more person and it will not be Donna. Jingles tries to kill Margaret but Xavier flexes his Robin Hood skills and shoots Jingles with five bows. Margaret kills Xavier, removes his ear and then Mr. Jingles wakes up to find Richard Ramirez who asks Jingles if he accepts Satan as his master and Jingles nods in affirmation. The sun rises marking the end of the longest single night in American Horror Story history spanning five total episodes and the school bus of kids arrive while Brooke and Montana are still brawling. Brooke repeatedly stabs Montana killing her as the entire school bus of kids watches. The cops arrive and Brooke gets put into handcuffs as Margaret tells the sheriff that Brooke went crazy and killed everyone. The paramedics find Ray and put him in an ambulance. However, as the ambulance exits the grounds of Camp Redwood, Ray is unable to follow. Montana recognizes that she's now a ghost and she kills the sheriff before finding Ray and Jonah and Montana tells them that they all can be gods in this camp Redwood purgatory. Then Satan pals Richard Ramirez and Mr. Jingles are able to exit Redwood in a stolen police car and they make it to Los Angeles. All right. It's 1985 and Richard Ramirez and Mr. Jingles are sharing a motel room in Los Angeles. Mr. Jingles expresses that he's tired of putting up with Ramirez's lifestyle and he's tired of seeing people die. But Richard says that there's no going back after Satan gifted him life. While stopped at a liquor store, Benjamin sees a newspaper with Richard's face on it identifying him as the night stalker. He hands the newspaper to a woman and indicates that Ramirez is in the store. A mob chases Richard Ramirez and beats him as Benji drives off. Time jump to 1989 at a now abandoned Camp Redwood where Montana kills a wandering duck photographer with Xavier's support. While the ghosts of the 1970 victims observe. Cut to an episode of the lifestyles of the rich and famous starring the real estate mogul Margaret Booth who has made a career of buying properties with dark pasts like John Wayne Gacy's home, The Winchester House, Spawn's Ranch and Briar Cliff Manor. And surprise, beside Margaret Booth is Trevor Kirchner who reveals that the knife missed his heart and he miraculously recovered and married Margaret. Flashback to 1985 when Trevor wakes up from his coma where he and Margaret make a deal to get married and share Margaret's riches in exchange for Trevor not turning her into the police. Back at the Booth mansion in 1989 and we meet Margaret's assistant Courtney who tells the happy couple that Brooke Thompson has lost her final appeal and is set to be executed. Meanwhile, the ghosts of Camp Redwood rummage through the photographer's belongings and Ray expresses his frustration with Montana and Xavier's killing habit. The photographer's girlfriend shows up and Xavier kills her to raise dismay. Also on death row, Richard Ramirez astral projects to Brooke's cell and offers her a similar deal with Satan that Mr. Jingles took. Brooke politely declines. Margaret finds out about the photographer and his girlfriend's murder at Camp Redwood which inspires her to plan a food and music festival on the camp grounds in order to capitalize off or in her words, baptize the campsite just in time for Halloween 1989. Now Benjamin Richter finds himself leading a new life in Alaska with a wife and a child. He is employed at a video rental shop and goes by the name Donald. Margaret returns to Camp Redwood and holds a press conference for the festival and Chet plots to kill Margaret as revenge for his brutal death. A visitor informs Richard Ramirez of the Camp Redwood festival with Billy Idol as a headliner and Richard says that he doesn't wanna die with the 1980s as 1990 is fast approaching. Richard plots a comeback so that his legend lives for multiple decades. As Brooke walks to her death, Richard gives her another chance to take his offer which she declines again. About to be executed, Brooke speaks through the one way mirror assuming Margaret is on the other side and pleads her innocence once more. Meanwhile, Richard channels Satan and possesses the prison guard who is assumed to have allowed for Richard to escape the prison. Back in Alaska, Benjamin finds his wife slain and a note on the back of the Camp Redwood festival flyer from Richard saying Satan will have his vengeance. Benjamin leaves his son on his sister-in-law's doorstep and vows to end the darkness that plagues him. Brooke's death sentence seems to be carried out until she is awakened and the executioner is revealed to be none other than Donna Chambers. It's 1948 at Camp Golden Star and we meet Bobby and LaVenya Richter, the brother and mother of none other than Mr. Jingles himself, Benjamin Richter. LaVenya scolds Benjamin for reading comic books and being introverted and Bobby wants to go swimming so LaVenya tells Benjamin to not let Bobby out of his sight and the two head off. After being made fun of by a couple of girls, Benjamin decides that he and Bobby are not going to go swimming. Then Benjamin sees the lifeguard walk off into the woods with a girl and he follows, leaving Bobby alone on the dock. Benjamin watches as the girl and the lifeguard do the deed while Bobby ignores his brother's directions and swims in the lake. Benjamin returns to the dock to find Bobby in the water as someone starts a motorboat unaware of the child in the lake. Bobby is struck by the propeller and dies. LaVenya hysterically tells everyone, including Benjamin, that they are responsible for Bobby's death. Now, back in 1989, Donna lays a recently revived Brooke on a motel bed and tells her that she swapped the drug that was supposed to execute her. Donna tells Brooke that she got her new name and a new passport, but Brooke questions Donna's motives and Donna tells her that she's trying to make things right. Brooke finds out about the Camp Redwood Festival and vows to return to Redwood to get her revenge on Margaret. To give Brooke a sense of what she missed while locked up, Donna takes her to a roller rink where they meet Bruce, who asks the girls if they can give him a ride to his girlfriend's house. Donna declines, but after he helps them with their broken down car, they agree to give him a ride. When Bruce tells the girls a creepy story, Brooke decides that his ride is over and Donna pulls over. Bruce refuses to leave and a cop pulls up behind them on the side of the road. Donna tells the cop that they don't know Bruce and the cop informs them about the slew of girls that have gone missing in the area. Bruce shoots the cop and exits the car as Brooke and Donna drive away. Benjamin makes his return to Camp Redwood and finds one of the Mr. Jingles impersonators that was killed during the 1984 massacre and the other two impersonators attack Benjamin for killing them. Before they can kill him though, the first impersonator stops them because Montana has ordered that there be no killing until the festival begins. The three Jingles bring Benjamin to the rest of the Redwood ghosts and Xavier and Montana tell him their plan to kill everyone who comes to the camp for the festival. Chet tells Benjamin about the lady in white, the mysterious ghoul that has been terrorizing the ghosts since their death. Benjamin tells the gang that the lady in white is his mother. Benjamin also tells him that there was a massacre at the camp even before 1970. In 1948, after Bobby's death, Levinia slaughtered the Golden Star counselors while they slept.  Xavier takes Benjamin to Levinia's shack and Levinia tells him that she's never been able to find Bobby's ghost and that she's been trapped at the camp since her death. Levinia then reveals that in 1970 she influenced Margaret to slaughter the counselors in the bunkhouse. She says that she used Margaret because she was the only thing that Benjamin cared about and she wanted to destroy him because she believes that the wrong son died that day in 1948. Benjamin tells his mother about his son and his quest to get revenge on Richard Ramirez. Levinia tells him that his son is better off dead than having to have Benjamin as his father. Donna and Brooke are stopped at a traffic light when Bruce slams into the back of their car with the stolen police car, knocking the girls unconscious. When Brooke wakes up, she's in the driver's seat of a truck and Bruce tells her that she either has to drag Donna to her death or get shot in the head. Brooke slams the gas and puts the truck in reverse, causing Bruce to fly into the dashboard. Brooke is able to grab his gun and shoot him in the leg. Meanwhile, Margaret, Trevor and Courtney arrive at Camp Redwood in preparation for the festival and Trevor finds Montana's ghost. Brooke and Donna tie Bruce to a telephone pole and sever his thumbs and Donna agrees to come along with Brooke to Camp Redwood. The band Kaja Goo Goo arrive at the campgrounds but so does Richard Ramirez, who kills the entire band. Levinia confronts Benjamin at the dock and Benji tells her that he named his son after Bobby. Finally softening up towards her son, Levinia tells him that his best chance at defeating Ramirez would be to die at his own hands and be bound to the camp forever. Benjamin stabs himself on the dock, dies and his ghost heads back to the camp knife in hand. On October 30th, 1989, Donna and Brooke discuss final girls at a diner until Stacey Phillips, a writer for the National Enquirer approaches them to tell Brooke that she looks just like the convicted and dead serial killer, Brooke Thompson. Brooke and Donna head back to their motel but Brooke is concerned that Stacey really recognized her. Meanwhile, Jonas wanders around the outskirts of Camp Redwood when he is picked up by Bruce who is driving a hot pink car. Jonas hears screaming from the trunk and it is revealed that a Mary Kay saleswoman tried to help Bruce after Donna and Brooke severed his thumbs and tied him up but Bruce instead attacked her, locked her in the trunk and stole her car. After she pleads with Bruce once more, Bruce stabs and kills the woman and Jonas disappears. Margaret discovers the death of Kaja Gugu and instructs Courtney to cut the bodies into small pieces and dispose of the band, which he reluctantly does. Benjamin confronts Richard Ramirez and the two brawl until Bruce hits Benjamin with the car.  Back at the motel, Stacey comes to Brooke and Donna's room and tells them that she knows who they both are and she knows that Donna was Brooke's executioner. Brooke tells Stacey everything, including the fact that Margaret committed both the 1970 and the 1984 massacres in hopes that Stacey will keep quiet until Brooke can get revenge on Margaret. Trevor and Montana lie in a bed and Trevor says that he can stay with her at Camp Redwood to keep her company. Xavier takes Bruce and Ramirez to Benjamin's corpse, revealing his death to Ramirez. Benjamin's ghost stabs Xavier's ghost for some reason and Ramirez says he's going to go to Alaska to kill Bobby and since Benji is bound to the camp, he can't do anything about it. Benjamin tries to kill Richard but Margaret shoots Benji in the head and she recruits Bruce and Ramirez to kill every headliner at the festival. Ramirez agrees as long as they don't kill Billy Idol. Brooke and Donna give Stacey a tour of the camp and they tell her where everything happened but Stacey is skeptical of their stories. Margaret sees Brooke and Donna from a distance keying her in on their revenge plot. Montana, Xavier, Chet, Ray and the three jingles tie Benjamin from a tree and tell him that he can't kill Richard Ramirez because they don't want to deal with his ghost being trapped at Redwood with them.  He also says that Montana ignited his fire for brutality.    Trevor finds Montana and tells her that he'll kill himself so that he can stay with Montana without growing old. Montana says he's too good for her and she won't let him do that. She tells him that she never wants to see him again. Xavier and the jingles impersonators stab and torture Benjamin on the dock and leave him to bleed out on a boat in the middle of the lake. Benjamin apologizes to Bobby once more and is dragged into the lake by a deformed swamp creature, Allah, Friday the 13th. Benjamin wakes up to find Levinia and his brother Bobby having a picnic. Levinia comforts Benji for not being able to save his son saying that nothing can rise above the darkness of Camp Redwood. Levinia says that she's eternally grateful that Benji brought Bobby back to her and she tells him that they can be a family at peace again if he stays with them and Benji goes off to play with Bobby. The year is 2019 and grown up Bobby Richter hikes to Camp Redwood in search of his father who he believes has been sending him money over the years. The camp appears to be abandoned since the 1989 festival. He meets Montana and Trevor who tell him that his dad is dead. Flashback to 1989 where Trevor blocks the roads to Camp Redwood and tells visitors and performers that the festival is canceled. Margaret shoots and kills Courtney for telling Trevor that Kaja Gugu was murdered and Trevor tells Margaret that he's divorcing her and then Margaret shoots Trevor three times. Dying right outside of campgrounds Trevor desperately tries to crawl through the entrance. Brooke helps carry Trevor through the entrance and Trevor dies at Camp Redwood. Trevor's ghost kills Bruce and throws him over the fence so he won't be tied to the camp.  They establish a kill watch system so that they can keep killing Ramirez at every revival. Back in 2019, Trevor and Montana tell Bobby that the reason they keep killing Ramirez is to protect Bobby. Chef Birdie and Chet are on Ramirez's death watch and Birdie flirts with Chet. The two make out and are distracted when Ramirez regains life and he gets away. He finds Bobby, Trevor and Montana. Trevor gets stabbed repeatedly while Bobby and Montana run into the woods. Bobby runs into Ray and Chet, Birdie and Montana overpower Ramirez while Bobby runs off. Richard Ramirez throws a knife in Bobby's back in the two fight, then the ghost catch up and fend off Ramirez while Bobby escapes. Montana tells Bobby to go to Red Meadows Asylum and find the medical director. After being apprehended by security at Red Meadows Asylum, Bobby finds Donna, the medical director. Donna tells Bobby that his father is innocent and that Margaret was behind the both massacres. Donna tells Bobby about the night of Halloween 1989 where the ghost pulled up to Margaret's cabin. Donna attacks Margaret, Brooke pushes her into a mirror and Margaret shoots Brooke. Trevor chops off Margaret's arms and shoots them over the property line using a wood chipper.  Back in 2019, Donna says that she was the sole survivor of the night. Bobby tells Donna about the money and they travel to Pineville, Oregon to trace back the sender. They find a middle-aged Brooke Thompson who apologizes for not telling Donna that she survived. Brooke says that the thought of Redwood was too overwhelming and she needed to forget Redwood in order to move on. Flashback to 1989 where Ray takes Brooke to die off the Camp Redwood property lines. But someone from inside the camp called an ambulance and Brooke survives. Back to 2019 where Bobby returns to Camp Redwood again and finds Margaret's ghost who says she can take him to his father. Margaret says she died before she got put into the wood chipper and she's been hiding ever since. Margaret tries to kill Bobby but Benjamin appears and stops her. Benjamin stabs her and then takes Bobby to leave before she comes back. Margaret stabs Benjamin and begins to chase Bobby. LaVinia appears and tells Margaret that Bobby deserves a happy ending. LaVinia then stabs Margaret and Bobby escapes the campsite and looks back at the ghost of his father, grandmother and uncle looking back at him. The end.


فشرف بالاشتغالي بالعلم ولا تبغي به ما عشت يدى بدلى وياله من شراف العظيم الموضوع الآخر الذي يظهر أنه يجب أن نهي على المكر من صفات المؤمنين هو كتروسيسة المؤمنين أيضا والمؤمنينة والمؤمنات وبعضهم اولية وبعض وأنهم برؤين مئل ومن رناسهم بمشاكات واحدة كم هل يشاركون عن جميعهم ذلك؟ يعملهم بالمعروفي يemphasهم لهم ويخبرهم بأنه كان جهل اللي ستبقى علىهم وانه ماسي. إن الله يجب أن يكون عزيز. اللي يكون غالبا في أمره. ماذا يريد؟ يجب أن يحدث. حكيمه وانه يكون عزيز. ثم اللي سبحانه وتعالى. يجب أن يجعل أمر بن معروفا النهي عن المونكر. يجب أن يكون لديه رسالة في مجموعة وفينا. ثم بعد في الأن اللي يقول أولائك سيرحمه من الله. هذه أولائك الذين يبقى علىهم وانه ماسي. ويجب أن يكون رحمة في هذا العالم ومن هنا. ثم اللي يقول إن الله عزيز وحكيم. اللي يكون عزيز وانه يكون حكيم. عزيز وغالبا في أمره. ماذا يريد أن يحدث؟ حكيمه وانه يكون عزيز. سبحانه وتعالى. لذا اللي يجعله أمر بن معروفا. أمر بن معروفا. من هذه الأمرين التي يوجدونها هو أمر بن معروف وانهي عن المونكر. وما هذا؟ ما هو the difference between the believers and the hypocrites. لأن the hypocrites are allies to one another and they prohibit each other from the good and they enjoy with each other the evil. as Allah said المنافقون والمنافقات بعضهم من بعض يمرون بن مونكر وينهون عن المعروفي. المنافقون are the hypocrites the male hypocrites والمنافقات and the female hypocrites بعضهم من بعضهم some of them are from the others meaning they are together for each other but Allah didn't refer to them as allies to one another. يأمرون بن معروف they call to each other the things that are bad and the things that are evil. so Allah said يأمرون بن مونكر they call each other to the evil acts وينهون and they prohibit each other from what عن المعروفي from the things that are good and then this is also the characteristics of the hypocrites then the mu'min should have a characteristic that is opposite to the characteristics of the hypocrites. also Allah SWT told us that the characteristics of أر أمر بن معروف من خسار الصالحين is also from the characteristics of the righteous people as Allah has said in the Quran ليسوا سواءة they are both not the same من أهل الكتاب أمه from the people of the book are a nation قائمة who stand يطلون آيات الله they stand and they recite the verses of Allah أنا and lady all day وهم يسجدون whilst they are in a state of prostration يؤمنون بالله they believe in Allah وليوم الآخرة they believe in hereafter ويأمرون and they call each other بالمعروفي to the good وينهاهم الله سبحانه وتعالى they prohibit each other عن المنكري from the evil acts ويسارعون في الخيارات and they hasten towards the good وأولئك هوم الصالحون وأولئك من الصالحين and they are from the righteous people so Allah referred to these people who have these characteristics from them to be the righteous people also Allah سبحانه وتعالى He told us in the Quran that the people who don't call each other to the good or who call each other to the good and they prohibit each other from the evil that the good of Allah سبحانه وتعالى will be established on them and this is how they will get the خيرية the goodness that Allah will give to them as Allah said in the Quran كنطم ورلي you are خير أمهة in the best of nations اخر جتلي ناسي that will send out to the people you will be the best but the condition is that this is found in you what is it تأمرون بالمعروف that you call to the good وتنهون علي المنكري and that you prohibit each other from the evil وتؤمنون بالله and that you believe in Allah سبحانه وتعالى and then Allah is telling us that these characteristics once it's found in you then what is found in you is what is found in you is خير and that Allah will make you the best of nations but when this characteristics is missing and it's not found then this characteristics of خيرية will also be missing as long as this is missing and then the condition will only make the thing that's been condition found when the condition is found now so what is needed here is that people know we will not be the best of nations if this characteristics is missing you will find some Muslims who see evil acts and they are silent about it they don't want to speak about it or they see an opportunity to talk about something that should be done and that Allah سبحانه وتعالى has made it mandatory instead of going out and talking about that thing he finds in himself and lack of ability to do so and this is a sign of weakness in the e-man of that person don't be a person who sees the chance to talk about something that is good or the chance to prohibit a person from evil and that you don't do it then there is a weakness in your e-man and insha'Allah we will speak about the things that or the times that and the different cases and scenarios in which it takes but quickly going over it the the legislation of of as I mentioned before the call of of it's something that there's no خلاف amongst the scholars that it's something that has to be done الإمام أبو بكر الجصاص he says in the Quran أكد الله تعالى أكد الله تعالى فرض الأمر بن معروف النهي عن المنكر في مواضع من كتابه وبيّنه رسوله صلى الله عليه وسلم في أخبار متواترة وأجمع السلف وفقهاء الأمصار على وجوبهم الإمام أبو بكر الجصاص says that Allah emphasized on the mandatory of الأمر بن معروف النهي عن المنكر many places in the كتاب and the messenger صلى الله عليه وسلم clarified that in narrations that have come to us in a large scale of number وأجمع السلف when also the highest predecessors they also have a consensus amongst themselves and also the jew spirit scholars in all of the world where they lived they also had a consensus amongst themselves also الإمام النهي we said قد تطابق على وجوب الأمر بن معروف النهي عن المنكر الكتاب والسنة that it has gone hand in hand that the man the أمر بن معروف is mandatory and prohibited each other from evil being mandatory has gone hand in hand with the كتاب of Allah and the the messenger صلى الله عليه وسلم and the consensus of this and it's also from an advice which is from the religion so you see we have how many scholars about ابن حزم and نووي that we quoted these three that doesn't mean it's only these much rather there are more scholars that have spoken about this Insha'Allah another time if you get an opportunity we will seize it Insha'Allah and talk about and talk about the the ruling that comes in different timings sometimes أمر بن معروف النهي عن المنكر becomes حرام sometimes it becomes سنة sometimes it has it has rulings why لأن الحكم because the ruling of any aspects in our religion يدوروا مععلي يوجودوا رعدنا because it revolves around the scenario and the place in which you are at there are times if you speak it will be more of a harm then it will be a good so in this time it's better that you don't quote it and the benefit in it الله سبحانه وتعالى knows best الله سبحانه وتعالى بحمدك أشهد الله إله إلا الله


Okay. Thanks everybody for joining today. This is the CNF working group weekly meeting. Thanks for joining. Things that will be going over today is the process and the structure for ideas to become adopted best practices and kind of jump into some of the issues that we have. And so the read, I guess before we jump in is there anything anyone else wants to add to the agenda. Okay. This is sorry this is Ruben speaking. So quick question. So the goal is really to discuss how to turn lots of the discussions we had into something more actionable right. Yeah, that's correct. And then that fits. Yeah, so I guess Ruben to jump right off of your comment is, I think what a lot of people have been seeing is that there's been a lot of conversations flying around both in this meeting on the slack channel and various different ways but it's been a little bit hard to make them actionable and in terms of how we want to move forwards with them into kind of like proposals for best practices. So we do have the, like, the best practices like, like, or like proposal template, but as so far, no one has really like open that open that up and so what we're thinking of the actual process is kind of like providing like a smoother onboarding and we're just kicking off discussions here in the CNF working group discussions board, and we can actually see a couple different, like topics on here. And so best practice like CNF should not rely on local persistent storage for data retention, best practice keep application configuration outside of the image. There's probably a couple different discussions around like defining the telecom actors. And so the idea here would be that first to start the discussion in a more informal way around the best practice is we would start start a discussion thread. And so what this would allow us to do is to basically have a written way of to look at the different best practices that we have rather than someone to have having a complete idea formed. And what this allows us to do is to kind of iterate on the idea before we need to come up with a more formal proposal. And so, in this way, all the ideas around certain best practice can be kind of contained in one place. The group thinks that there's enough information discussion around a certain best practice, then a more formal proposal can be made. So if you think about it, kind of like the, the proposed process right now would be first, like either would be starting with something on this discussion board, creating discussion discussing it in this meeting, generating enough ideas and interest, and then the next step would be actually defining a actual proposal, and in this way, instead of having a whole bunch of conversations in Slack that you'd have to search through each best practice kind of has something behind it. And then obviously the final part would be reviewing and adopting the best practices. So, kind of with that in mind, what do people think of this newer kind of like less formal process. Oh, the, just to add a little bit to it. So the, I don't know that every, every topic in here is going to become best practice, especially ones that are meant to be like defining the talk one actor so any, anything that we're discussing is going back and forth and we keep asking, let's try to get it in so that everyone is getting those seeing the conversations, whether those are on the meetings or in Slack, it's fine to keep having them. Of course, any, any area that try to capture something so that we can have that shared understanding. That one of the main reasons to get it in here besides making sure that everyone can see it is to make the hurdle for, I guess, sharing the ideas broader is the formal adding a proposal may feel like you don't want to do it yet because you don't have all the information so to lower that hurdle for that. And of course, if anyone see something that they feel like there's enough content. I think that we can put something forward we have all the references we have the use cases we have all the stuff that we feel is necessary to communicate that this is a best practice that we want to put forward, then open a PR. Do a PR an issue, put it forward. We'll still have discussions just because it comes out of here doesn't mean we'll stop. It's just lowering the bar so that we have a shared area. Hey Taylor bill. Hey, so one thing I would like and it's going to be contentious and rough. But I think it's necessary before we start defining best practices is we put in our charter that one of the things that this group would do would put forth our definition of what a CNF is. I think that's really important. I know I've talked to you a little bit about this Taylor but I'd like to, you know, put it in front of the group is if we don't have a definition for what we're proposing best practices for. There's a lot of ambiguity that can get us into trouble. For instance, like, you know, there's best practices around the amount of privilege that a container gets, and we could get into specific, you know, nuance discussions around least privilege and the container having exactly how much privilege it needs.  If we don't know like whether or not the orchestration for say the infrastructure comes along with the CNF itself if we don't say that you know those are two separate things. And some CNF developers put their orchestration pieces in with the CNF among it all together some don't begin this weird thing where we say it's a best practice to not have, you know, privileged containers. And all of a sudden we're in a scenario where we have a container that's you know an orchestration platform that's been quote unquote labeled a CNF, because it was you know just bundled that way. And it's now you know breaking a best practice because it's provisioning block storage for me or something right like, so I think we need to make sure that we a define, you know, what we're saying this is a best practice for you know is it a single container, is it a collection of services like everybody has a different opinion, but it would be nice if the CNF CNCF gave us like a, this is what we think it is. And then from there, we can provide context in these best practices to the type of workload that the best practice applies to is.  So we can just make it, you know black and white by saying this is for the CNC. This is for a CNF, and we've excluded these types of containers from a CNF, and then therefore you're not breaking something with the best practice like, you know we're calling this an orchestration container or something. That sounds good. Jeff, I think that there was something that you talked about last week. In chat about exceptions to so that context so this would be tying in with what context do we need to know about a best practice. So that you know how it applies. And it seems like you're related. So, so yeah, context, how it applies and then I mentioned exception and then I'll let Ian speak to it because Ian was actually talking about like a formal process he's used in the past but the big thing is right is we don't want something to instantly be disqualified if it's a really good idea because it breaks a current best practice and maybe that best practice wasn't exactly supposed to fit to what that specific container was trying to do so. Yeah, exceptions in context I think are super important, because if not we end up with these really broad brushstrokes that either limit what we can do, or at least things so open to interpretation that you know we could both be quote unquote following a best practice and have completely, you know, polar opposite results. I mean, when I did this in. Well, it wasn't say it was safety mitigating which is a little bit of a get out clause for the companies I were working for but safety mitigating software then what you would tend to do is you would have. If I put it in the words we're using here you'd have a set of best practices and then you'd have a baseline of these are the best practices I'm applying in this particular circumstance, for whatever reason, not necessarily all of them. And it meant also that you could potentially have conflicting best practices something was a best practice in the circumstance but not generally, which meant that you you comply to a baseline which would be some subset of your best practices or maybe a bit of context about how they were applied, which meant that you didn't have to get it right. Well, among other things for us, it would mean we could evolve over the course of time the best practice list would not be a list that we couldn't change because it was already in use we could come up with a V2 or a context specific list for certain types of application, or certain circumstances. And I'm thinking about this. Because, you know, it's nice to think here we're in a relatively academic environment where we're talking about stuff that doesn't change the world. Or alternatively the world will just naturally comply to it whichever way you want to look at this. But we know that what's going to happen here is that whatever we write down ultimately somebody's going to throw at me or another vendor as an RFP. So we have to ensure that effectively it doesn't stop us doing the best job we can do because there's an awkward bit in it that just turns out to have been not a good choice in in practice. This seems like the way to make sure we could potentially evolve so that the bits that become, you know, turn out to be practically difficult, or not as valuable as they looked like we can throw away over the course of time. So the idea that we have best practices. And also we make a baseline that says these are the best practices that get you to, you know, a 2020 gold standard CNF. If we keep those two things independent of each other we'll get I think a little bit further, which can also be read as saying, you know, if you've got a best practice that you think is, you know, the best we can do today, as opposed to the best we might ever do, or alternatively a best practices that doesn't apply in every circumstance all the time. That's still not an excuse not to write it down. You should write it down put it in, and we will find the moment to apply it and if it turns out it gets outdated, we will have a means of saying actually it turns out we won't be doing that long term after all. We can get more ideas in there and we can choose our baseline it's not we haven't written something in stone it's such that it can never, it can never be altered. I'm just trying to capture what you said either that's a horrified silence for everybody else or you will think I'm absolutely genius and you're speechless I don't know which of the two anyone care to come in. I was wondering whether you could recommend because I think we just have to start and do it. So my question will be more whether it is whether there's any any recommendation to start codifying this this basically coming on the one hand from the definition of a CNF. And on the other hand from codifying how to come with best practice and probably will end up meeting in the middle without chopping off our respective ads. Yeah, I think. I mean, Taylor put something up which I did like the definition of audiences which is context. And we definitely want to get that context written down, because best practices relate to context you can't have a best practice if you don't know what it's a best practice for. So we have to approach this from both directions at the same time and then that baseline will be saying, you know for our understanding of context for this list of best practices we recommend these ones right now. So effectively, three tracks. One is the context behind the CNF how it's going to be used who's using it what matters to them, and what audiences we have. One is a set of best practices that are probably more related to the software engineering practices of working with cloud native applications and using containers to implement them. And then the baseline that we'll eventually put together which is going to be the standard we release and tout to everybody and say you should be compliant to this. And then there is where to start or where to focus. And my view is wherever you have, wherever you're reading something and it seems to affect you as probably, or whatever's interesting. So we don't have to stop, start with any specific ones. I, some of these like this actors is going to affect everybody but so please at least review it. What a CNF is but maybe we should start with what a CNF does before we talk about how it looks like how it's supposed to behave what are the best practices around it. What does a CNF do what qualifies as a CNF Indian without putting in a definition for for the world but what qualifies as a CNF in our view. And maybe we can, we can start from that because once you know what it's supposed to do you can start working towards how I think that was Christian bill and I agree. Like, I think that ties into the, the requirements thing right like, what do we need us like, not just what does a CNF do but what do we want it to do right like, why am I sticking this in my network. Because what I don't want to hear is that a CNF is a VNF that follows cloud native principles. Like, and I've heard that, you know, spoken as can and more times than I can count but like, you know, Christian hit the nail on the head, like, you know we keep coming like why are we doing this and I mean we all know what we have to make this cloud native transformation so like, we get into the whole motivation discussions I mean it's common vendors are making it this way service providers are deploying it this way like it's just whether it's a chicken or an egg scenario it's happening so like, we need to start identifying like our requirements on like how and why we're doing stuff right like, what is this CNF supposed to provide us, you know, from a developer standpoint from an operator standpoint, all those people that are in the actors page like, we all collectively are saying this is the path we're doing. And, like he said, what does the CNF do. And then from there, we can start to describing, you know, what it is. Since we're talking about network here, can we summarize it as CNF moves packets around from a to be with that qualifies a very, I don't know 10,000 foot view of CNF. And the data plane though what if it's like a control plane or a management plane portion of it, it doesn't move packets. How do we modify that. I don't know, but I'm just pose playing devil's advocate. Well, that's why I asked the question and I was just reading through the monster discussion that button chain we're having on the subject of, you know, the fantasy that I think we all subscribe to we all wish we were there is. Obviously enough from whoever supplying it with it's a vendor or it's another team in my organization, it really doesn't matter the point is it's an independent unit that's not me. And I want to get it on my network in 24 hours. How am I going to do that. The practical problem that you've got there is that you're integrating it in a wider system and you would like to make sure effectively all of its interfaces work the way they're supposed to do all of those touch points. And then you actually meet with the expectations of the thing that's consuming them. And then you'd like to put it through its paces as much as you possibly can so it's got to effectively be easy to test, which means it's got to be in turn easy to deploy easy to upgrade this sort of thing. And for me that the way to determine what requirements exist is to actually put this in the context of problems like that this is the thing that I would very much like to do. And I'm assuming that even if I don't get to, you know, a 24 hour deployment cycle or a two minute deployment cycle, whatever it happens to be, then at least traveling that journey will give me something that continually gets better. And what is going to help me on that journey. And then you get to a definition of what I'm looking for from a CNF and then you get to best practices in the CNF. But doesn't that differ from one type of application to another. I mean, in the end, why would you want to put a minor change in your network or a new application in 24 hours. Yeah, fine. But 24 hours one second the point is fast right. I have a critical security update I want it on my network, and I don't want my network to be less stable as a result. That is effectively your aim. Isn't it. I mean, is there any case where that's not true. Well, you do that in basically when you say 24 hours for an application that I want to patch. That's lifetime. If I want a new application that I have no idea what it is and I want to put it in my network in 24 hours, then I do have a problem. But if I want to patch something, there's two different workflows here. One is patching so rolling out an upgrade in 24 hours, which is fine. I think we're beyond that and we should target for a lot less but fine with it. Don't get tied up with the number. The number was just the important thing about the number is it's less than three months, which is not an unusual length of time for a physical device. So I'm just picking a number out here. That's all. Of course, the point I'm making is there's two different workflows. One is patching, which is driven by some considerations like security like compliance like bug fixes and the other is new deployments. So when you do have a patch, you're kind of assuming that you have that application in production already and you're figuring out the qualification mechanisms and upgrade mechanisms that should pretty much be allowing you to do this in line in place without any sort of service interruption or service interruptions and the questions are around how do I qualify this against my previous baseline? What do I need to test? And what ensures that I can move forward in the steps? And what are the checks between various canaries that I'm going to do until I reach a full out rollout? And then if for some reason that fails somewhere in the intermediate steps, how do I roll back? And how do I ensure that everything rolls back cleanly? Now, when you're talking about a new application, that's a completely different workflow because you have to understand how that application integrates into the wider ecosystem. And that study and the definition of those, I was just looking over one of the discussions and one of those is around dependencies. You don't have to worry about them or you have to worry about them in a different way when you handle an upgrade, whereas you have to worry about them in a completely different way when you have to instantiate something new. In one place, you want to ensure that you act on the components in accordance with that dependency so that you don't break the dependency chain and you end up at some point causing a service outage. You want to identify what are the dependencies, who's going to, what am I going to consume with this CNF and what am I going to provide the CNF. So the two workflows are in my mind at least are radically different. I don't know whether they're radically different because you've got to be at the end of the first with the workflow you're talking about when you've eventually worked out how to deploy it for the first time. You've got to be certain that you're ready for that second workflow that the short duration upgrade. Of course, you've got to, you know, it's not just about getting the application running in in your network. It's about understanding that there's going to be another copy of this application turns up, you know, next Wednesday, and I'm going to want that in my network as well. So at the end of the first process, you're going to have to be prepared to start from the second. So it's not, perhaps we're not comparing light with light. They will absolutely be different, but the first one has to end at a point where you're prepared to start the second one at any moment. So, okay, we want to tackle both at this stage or should we pick one like let's pick the incident one let's pick the one that we start with. So if we can model that and understand it and then we move on to the second one which obviously has slightly different behaviors and maybe we'll understand something else after we model the first one. That was the original. Yeah, no I see what flow or should we look at the instantiation workflow first. What I've been saying here is a specific workflow and you must follow it. I think what you're talking about the needs behind those workflows is more important because we're not saying we're going to copy over what's there and add to it. What we're looking at is, if you have something like we need to get a security patch out. So on the life cycle management or you already have something out a application running, you need to get a security patch out. So what are best practices that help with that that one goal so that's one thing that you want. You have something else where you say a misbehaving app is running and we don't know what the problem is so what are best practices that help you observe and get information about that application best practices on Kubernetes. So those are the underlying concerns so whatever level you're at a new deployment, a running deployment, rollback all of these things, what are those underlying needs. And now we can start looking at what are best practices that already exist in Kubernetes that are applicable. These are all ones that we should invent because we need them but all I would say is there isn't necessarily anything stopping us from writing both of those together, but what you're saying is that what I write in one is necessarily going to have some effect on the other. I don't think that stops us making the attempts simultaneously we just have to understand that the first try at this, you know that there's going to be a revision process it's going to take some time to get to a place that we like. I would write both of those and it was what I was saying earlier this is these things individually these are not best practice these are context effectively these are the things we would like to do this is the context that will determine what best practices matter to us. So it looks like we seem to be going all over. I thought that Jeff started the discussion with the CNF what is the CNF. And now we are we are going into all kind of workflows and the best practices for for workflows and whatnot. And we completely drifted away from what I see enough is and and the, the point which I wanted to make for a while ago, as soon as Jeff started a discussion that was Taylor, during the time. Didn't you define or did didn't we define those CNF where we call them bronze silver and gold and whatnot. And there was an attempt to make that we're what what these CNF are what do they look like what do they have at least, at least an attempt was made there. So where we're not picking up from there why we're trying to reinvent all over again. There was I mean, so people still will I mean I said into a lot of meetings, and I hear people argue about it all the times that guys VNFs are not going away. They are here they're going to stay here they're going to stay here for a very long time, and, and vendors need to see an incentive to throw away their VNF implementations and start getting into CNF implementation with not even knowing what a CNF implementation looks like. If I, if I get into room I bet you right now we have a 40 people or 30 people whoever we have on the call. If I asked, you know, that what is a cloud native, and we're not going to get one answer we're going to get at least 20 answers. So, and I do that all the time. So why are we, why are we not starting with there and I thought tug was doing that. Tugs, tugs started to do that, especially when they got into gold CNF. No, so they were bronze they were silvers, but the gold CNF gold CNF is pretty darn close to what we are trying to look here for what a CNF should be. So, so why do, why don't we start with that. And then, and then at the top of the hour when Jeff started he started with, with CNF, and now I think we've gone to security patch installing and we've gone into upgrades of workflow. So what, what is the goal of this group. What, what are we what breakfast practices are we at the end of the day, we can meet every week and we can talk all kind of different things and we can even do whatever we want to talk. And by the end of the day we have to set up something out of a goal that what what we want to achieve, so that we can set up some time frame, like by by q one end of q one, we want to be able to achieve this, and by end of q two, like Robbie is on this call, who was leading CNPT. And the only reason we made tremendous progress there was that he will step up and he will start setting up the goals. You will say, Okay, guys, let's define this release. Let's define this content. Let's go with this and work with that. So, so here my suggestion is, we should take some kind of similar approach. Right. And let's start writing it down. Let's start defining it. But then, then we take it as as it comes rather than because everybody seems to be wishy-washy everybody seems to be going all over depending upon what our personal preferences are what our backgrounds are where we come from. Right. So, so there's a lot of bias based upon what what we have been exposed to what we've been using to, but but there are certain things which are in front of us. So whether we pick what is available and then start to work from there. And my suggestion is that there are write ups on on what what gold CNF are. Let's take a look at it. Let's review it. Do we agree? Do we not agree? You know, if we don't agree, why we don't agree, let's fix it. Let's call it as a CNF. And then, then we've communicated on the goals just to I agree on the tag side. We've done a lot of discussions about all of this. There's a lot of information. Geoffrey's posted something there. That's now that's what's used in the text. So that's kind of the starting point for all of this. The principles to go with what you were saying as before. Yes, there's a lot of people that will say a lot of different things. But if you go and look at the I'll say the experts in the field. Whether you're saying specifically cloud native, whether you're going into Kubernetes usage, whether you're going to stuff that builds into cloud native dev ops and wherever you want to look. There's a lot of content that go right into what the principles are about. It happens that CNCF doesn't have those expanded out directly in their definition, but it doesn't mean that the experts that put forward these haven't said a lot. So this is a good place to start if we want to look at some of these things, but I don't think Geoffrey was specifically looking to say, do we have any idea it was more of are the best practice that we're putting forward going to be applicable or are we going to cause problems. So it's a more specific thing. And then as far as the goal of the group, we've kind of communicated this again and again it's in the scope for the charter. We're looking to determine what are these best practices that are going to allow the telco applications to most efficiently use Kubernetes. And they have, why are we doing this? We want to lower OPEX primarily, I would say, but it probably affects CAPEX. We were talking about life cycle management. So how are we going to lower the OPEX cost, whether that's reducing risk, whether that's making it easy to manage changes on the network because you push more stuff out to the infrastructure that already is handling some of those things. So if you're building an application that you're selling to a SP, you know that you're focused on the part that matters the most that you have the expertise on and you're not having to do additional work outside. So it's all around those same things as far as the reasons, but the goal, that shared goal of how to best utilize Kubernetes, the framework, the community. So outside of just our group, how do we bring more and more in that already have the knowledge on running these applications. So in effect, the goal is identify best practices. You just summed it up. It's not defining what the CNF is, but we kind of settled the CNF is what was defined in the talk and the goal of this group is just to identify best practices. And this is something that we kind of went all around the place before. But that's fine. If that is the goal, then yeah, we can focus on that. Where is very clear stated that is where is the tugs definition of what a CNF is and how extensive is it. I think it's right there in what Jeff put. And so, I think to both Christian and and the what the reason why we're bringing up CNFs a what is a CNF or what are the goals of CNF or all of these things are because we need context the same same as everything else we're talking about during this call for why a best practice would be put forward it at all. How does this affect me, is it going to cost me problems, is it going to benefit me. So it's all context so we do need the definition, and probably it's pointing at it so in, I can't right now drop a link and that says here is the type definition which is a problem. We should make sure that that's very easy to find on the tag, and any type of context and content around it, we have all of that so that'll be something that I'd like to put on at least my plate and if anyone wants help to look at. And then we can do the same thing from the CNF working group where we go. Here is what we're referencing. Along with what Jeff is saying, but what we probably don't have anywhere else is some of the context that Jeff was worried about where we put a best practice forward. And it doesn't apply to a CNF. That's very common so we have something out there, we put the best practice, it only causes problems if you use that best practice on the CNF. So how do we communicate that that it's not either not applicable to this one you shouldn't use it, wherever, however we deal with that. I believe Jeff the main thing that you're wanting to get forward on this is, it's not applicable. You got it spot on. So I just confirmed my own fears. I did not speak with enough specificity with my initial discussion thing and so then lots of room for interpretation which mock things up. And I think what Jeff said on Taylor's like, would a CNF that's specifically targeting layer two, and potentially, you know, interacting at like the Nick layer going to have the same best practices is maybe one operating at layer three, and is you know looking at rap tables, etc. I don't really interested that right now just hypothetical right silly. It's exactly what you said is, you know, do we expand upon it, and ensure that we, we just give like the proper context for when we say best practice of like, this type of workload because it's going to give you the optimal results based on what this specific, you know, container function, whatever it is trying to accomplish. And then, if we need to expand definition, we can, you know, if we need to contract it whatever but like the big thing is if I just say no, no root, you know, privilege in any containers. Well, is that always the best practice. It might be that the cases it is it might be the case that it isn't and I just want to make sure that I'm hoping that this is a little bit more lower layer than like the more abstract stuff where it's like here's a reference architecture. You know, here's a white paper like my hope is is, when I come from this group I have things in my toolkit that I can actually go and you know pun fully intended put into practice. Versus you know just abstract concepts that I'm like okay you know these are great guidelines that I'm going to follow it like follow that like a broad level actionable versus just thought something to think about. Do you have something concrete or is it abstract. Yeah, exactly. So for looking at an example. Okay, specifically is this is this like the type of content that you're looking for like CNF container should run on privilege. Yeah, absolutely right and once again though it has to have the context because Ian's concerns are valid. I'm going to put this in an RFP, you know, are these containers unprivileged or are they going to try to mess with my, my infrastructure. Are they going to try to talk to the kernel and do things that I might not want to do. If we have the context and I would try to tailor that RFP to make sure that once again we're calling a least privilege model versus a no privilege model. But you know this is something I can test to like I can go in, and then I can try to like go into one of these containers and try to execute system commands that I shouldn't be able to do I can test this. And then I come away with, you know, a slightly more warm and fuzzy that if I deploy this container, I don't have to worry about it going in and doing things that maybe I don't want it doing in my infrastructure so I do think that you know discussion number 25 here is a great example of the type of thing. I personally and maybe not everybody feels this way is trying to get out of this endeavor. Another thing, again that you can, we may be able to steal from my past life is when we were doing compliance in those days, then there were a set of rules, largely around right software that, you know, doesn't break in horrendous ways and kill people. And there were a set of exceptions you could, because no one's assuming that a rule applies blanket across the board perfectly, you could absolutely write a set of exceptions but there was a formal process for writing the exception down, explaining the exception and supplying it along with the application you were shipping. And, yeah, it's a get out clause. It basically says I can be compliant without being compliant, but on the other hand, if we had a best practice along the lines of this is how you document why you're not following a rule in certain circumstances, how to identify that how to demonstrate its and so on and so forth. And that also might work for us. Because Christian right there on the screen and put in an example of a perfect idea where like, if I'm, you know, and this gets into the whole other best practices on the dev side might come out of like do I compartmentalize. So there's not one giant homunculus container that has all these things because it's doing like layer one and two stuff it's doing layer three stuff but you know he puts like thing like if we decide we're going to go down an SRROV path, I spin up a new container. It's going to have to, you know, be able to interface with the system and pull that the FID into it etc right so that's Christian put like a great example of the types of stuff I'm just curious and like 99% of the time, it's going to be applicable but here's an example of where it's not so we document it as an exception and we provide the context of what it is a best practice. One other thing that for me personally is a pain point is the dependencies. I think Jeff started as discussion or it was started from a thread and slack, but one of the biggest issues I have when looking at a CNF or a VNF is what are his dependencies who does who needs to talk to this CNF who needs to consume it. What happens if it doesn't. How do I ensure that its dependencies are before I started. Today we use a very wacky work around using staconities from OpenStack. And that kind of, I mean, most of the time that works, but when we're starting to look at layer 2 VNFs that falls apart because it's not able to probe layer 2. So can we think of some sort of a generic interface is anyone else facing this dependency problem. And is it a real problem or it's just something that I have. Can I pull up out that word dependency for a second. Are we talking about system interfaces that all CNFs must consume as in that there's a common dependency. Are we just talking about a relationship between two CNFs or something like that. My idea here is more around relationships not interfaces to the system but when you have 235 CNFs on the same Kubernetes cluster, how do they know about each other. How can we do maintenance on one while not impacting you ever how can we suspend ones up operations or how can we let's take an example from the real life world. If my upstream link is down I want to ensure that I put the ports on the switch down so that I immediately take the entire data path offline so that my applications can fail over to the active one. Can we do something similar for CNFs. Can we inform one that its dependency went away. So it shuts down and commit shifts over to its standby. Christian, are you talking about a, I'm going to call it a product of vendor sends a specific type of application to handle things it may have many different components and parts. Do they all need to work together. It's more about multi vendor let's say I have one component from one vendor another component from another vendor and the upstream component needs to inform the downstream one. All right, that in a generic way, so that I want to swap vendors from both upstream and downstream and not have to worry about the fact that I'm going to need this interdependency to work and those vendors need to be certified. I want to, can I do my qualification 24 hours or not. If vendors respect that interface, then it's easier. So if you, yeah, and you've hit the nail on the head here it's not about the two pieces of code. It's about do they respect the interface. So you would, you know, I'll use BGP is a fairly traditional example here in theory, if not in practice, every BGP speaker you can pull off the shelf will talk to any other BGP speaker you can pull off the shelf. And you're already 95% confident that's true because they both talk BGP and you can check that they're compliant with the standard before you even get to talk to each other. And what you're talking about here is kind of the same thing. You want certain link up down behavior from a CNF, and it doesn't matter what's on the other side of that link, whether it's another CNF or your switch or anything else. The behavior is something you can document it's the way it must respect that interface. So something a bit more generic so not let's forget the network world but say my control plane goes down and I can no longer manage my data plane. I want the whole thing to go down so that if the control plane has a BGP session with an upstream data center gateway it can fail over somewhere else. Basically, let's think about this in a more abstract way than just BGP. For sure BGP can solve this at 95%. I was using it as an example. I wasn't sure. I mean I know what you're thinking that you could play with roots that's not my point that's not the point I was making I was saying that BGP is a defined protocol so rather than defining anything else what I'm actually defining is the place where these two potentially huge pieces of code touch. And because they touch in a very small area, you know the protocol that is BGP, then you can document the protocol and you can be almost certain. Before you've ever brought them together on the same system that they're going to work together because they both respect BGP and you can demonstrate that. I can do this with API is the same way. Theoretically, I'm struggling to think of a fine example of an API but the idea would be there is an API definition for a. All right, let me take a weird example but it will do for now right. There are many implementations of cron out there, but I don't have to know the details of each implementation because ultimately, they respect the same CLI, I use them in exactly the same way. So regardless of which quantum cron implementation I happen to be running on my box, it will do the same job if I tell it to do the same thing so all I have to document is how I use it know not what it's doing behind the scenes. And so theoretically if I write strict the programs cron and I and I install whichever version of cron I personally favor on my box. I know these things of 95% likely to work together because the API definition the interface is what I've written down. And so in your case you're saying are there some generic interfaces. I know interfaces are loaded term in networking but you know what I'm saying here. Are there some generic interfaces I could write down where specific kinds of touch point could be documented. I had in mind something a bit more it side. This is from Juju from Canonical, which has providers and the consumers kind of the interface. So it's a declarative way of saying my VNF provides load balancing services. And my web server is a consumer of load balancing services if I tied those two together I know that the load balancer will load balance traffic, coming into my web server. The perfect example of this of the proper implementation is the ingress controller. I know it's crappy everyone hates it and I'm streaming and wants to rewrite the ingress interface but it actually does what what it's supposed to it says the ingress controller provides this kind of service and I'm the consumer of the ingress. For sure one of the issues of the ingress controller today is I cannot tell from the ingress controller to the back end that the ingress controller is down. The back end has no way of knowing it's no longer receiving traffic except for observing the fact that it's no longer receiving traffic. But it's a it's going along those lines of declarative dependencies where I have one component offering a service and another component saying I'm going to consume that service. Yeah, it's a contract between the two exactly. That's where I was trying to get. We need to define this guy or at least in my mind I think we need those kind of contracts and as a community we need to agree on those kind of contracts so they can be honored by different vendors. Yeah, I think there are a few other really boring contracts that we need to remember as we're doing this the one that keeps coming up with me with my grot lot is logging right I know I absolutely am 100% certain that whatever application I write with it's a CNF or anything else it's going to spew logs at something and something's going to want to pick them up there needs to be a contract between the two so that way I send logs and the way it receives logs we both agree on how that's going to work but it's still a contract. Yeah, that's, that's a perfect example of logging is really painful today we in my case I have two types of logs. Basically, one structured logs and Jason structured logs and figuring out which goes where and telling log collector, this is Jason structure this is not easily painful. Today my log collector really needs to parse land by line figure out if it's proper Jason tried to interpret the Jason otherwise it falls back to just sending the message on parse. And this takes enormous CPU cycles when you're dealing with text. Just a question, would it make sense to start thinking about the table of content that need to be created for the artifacts regenerating I mean this discussion are great. I think, if we try to convert them into pro request instead of just discussions that might be better focusing on the attention effort into making something more tangible. So is there any table of content being created to define how the structure of the release will look like. So, the thing that we have right now is these proposal like definitions. And so this is similar to like a Kubernetes cup, where like each one would define like a best practice. And so we're just thinking of seriously numbering them but if you think there's like groupings or something else that we could have. Feel free to open up or press Robbie was there a specific idea that you had. I'm thinking, like to work backward from the delivery that would like to make now if we say, we're going to release something as part of the outcome of this working group what would that look like is that document with a certain structure, different sections. And if we say okay, if this is the overall definition of what we're going to deliver. I think going back to the discussion we had in the slack channel. In the last time call, we decided to look at the guidelines and best practices for the CNF developer himself. I think if we just focus on that and think about what would the content be look like and start with some real content in. We will be able to identify the relationship with other parts of the platform for time to think about what other areas we need to work on next. I think that is this discussions are great, but I want to be very clear and blunt here. I mean those discussions does not really lead into tangible outcome of something that's you can actually refer to and point that. I think it all starts in my mind if we create that structure on and document structure with the different chapter sections wherever it is. We can work in particular section of particular area, even if they can have an off all or something that for those want to see to work more into the technicality of that part to be more like tangible outcome that we can lead to. Probably that's great. We didn't go into all of that earlier I guess when we're talking about structure what's next beyond the process to get to a list of best practices. What you're saying is is needed so if we determine a set of best practices that are helping to most efficiently use Kubernetes. What is that mean so one part is what Bill is saying is similar to caps. So this enhancement proposal, but the other part may be more similar to the Kubernetes conformance program. So if you go and look at any new release where they say here is the version of Kubernetes and what tests are required for conformance. There are only tests, but you actually have the test and why they were there so why is that feature part of Kubernetes. That's not immediately apparent if you just look at the test you'll have to dig in, but that's a similar thing here. So if we end up with a set of best practices that you can go and look at. And in this folder will probably have something similar to what the Kubernetes conformance which is a large document which list all of the test that are required and then you can click through and go look at what those features are for us it's probably going to be. Here's a list of best practices in a document and some maybe a subset of information so you can quickly see here are the ones that we're recommending that people use and the audience and and just a small blurb about why it's good. Click here to go read in details about this best practice that would be similar in my mind also to 12 factor apps. So now we're just helping people do better. But beyond that how are we trying to be on the document like that which I do think is good 12 factor, if you go look at whether you're looking at 12 factor, you have a list you can go dive in deeper, and so on. Beyond that is the test suite. So the test suite is currently developing and working on based on feedback it's already getting so talking to vendors talking to Kubernetes folks looking at stuff that seems like good things to test, but one of the things it'll do similar to Kubernetes so Kubernetes is writing tests all the time. And then eventually they come out with the best practice or not a best practice they'll say this is a conformance so this is a required feature this is something we really think everybody should have for interoperability. So if we come up with the best practice, it may be a one to one to a test, or it could be something like the idea of least privileges for CNS, and then a subset where we go here is something to do that. So now we can say what are some things that we can actually test. So the test suite is primarily to allow CNF developers or we'd say vendors to go and run the test suite and see how well they're doing on these best practices that they think are also good hopefully so that they can improve. Maybe it's about storage data or how you store data and cloud native way, but those will be working hand in hand as a result of the best practices. So Robbie are you interested in helping created table of contents. Yeah, I'm happy to have a look at that yeah. Okay. That'd be great. And I'd love to have that discussion when you're ready to have it. I think other people would, I think welcome the structure in it too. In the meantime it'd be great to move forwards our discussions in the discussion board around the different best practices that have been pro so far on the definition of a CNC of a CNF. I know we're at time. You know, we are having one meeting and then we're skipping the next two weeks and starting again. Not on the fourth because I'm assuming everybody's just going to be back into the office, but our next meeting after that will be on the 11th. So, thanks everyone for your time today I really appreciate it. So there's a meeting next week and then meetings for two weeks. Okay. Yeah, exactly. So, thanks. Cheers. Thank you.