I've wanted to grow can to hear from late and sports cards and I'm ripping first break of the daytime We've got 2022 top stadium club chrome baseball. Hobby eight box break number seven pick your team Yeah, Billy Kay Yankees Christoph s twins Edward P cardinals Ethan s white socks and Mets fight LG with the athletics and Rockies hunter hd backs and Braves Jason P. Padres just G with the Marlins Joe T Mariners Phillies John a pirates John be with the Giants KP AC with the Angels in Rays Kevin D Astros Lisa B Royals Mark H Cubs Michael s Rangers Patrick T Red Sox. Thanks again. Patrick's gonna close out Paul o Brewers Robert G Dodgers Ryan B Nat Samuel G Blue Jays Thomas F Orioles Victor F Guardians Wayne P with the Tigers and Reds All right, here we go first half of the case. We got the second half up next still 16s to go on that Yes, by now all hardship out of this break. Yep. Yep The tiny hands pick it this year I did not Patrick So you got two gloves on to beam team Vladdy and first pack We've got our first auto and it's gonna be to the twins Jose Miranda Minnesota Nice looking autograph there on the rookie auto. Nice Schmidt and a wonder Franco rookie base Y'all have to look while who's off the look Trophy Hunters chipper Jones Supers looks so sick on this stuff We've got blue of Mark Colo Colo Savi that is 135 of $199 on the rookie Alonso and Hoy Park. I Love encased such a fun product Team of the future Jonathan India Gosman Grossman What's that come out Patrick probably fairly soon, right? You've got wave refractive rookie Chas McCormick Strobes Beam team wants so no and we've got what I say two autos in this box That's gonna be a green rookie a Bryson stop for the Phillies. There you go 32 of 99 You see that pretty often some of these boxes with an extra auto in it Love to see that Blade Dynastie Calvrick and junior refractor And we've got gold rookie is gonna be a Kyle molar for the braves that is 18 of 50 Nice photo there with the flag around them Love stadium club Photography is unmatched Oh nice Patrick. Yeah, I'm assuming that would be coming out soon. Oh Tanya team of the future refractor angels Story McCutcheon that short rookie got refractor Andrew McCutcheon Team in the future rookie refractor Julio Rodriguez Mariners Steele Korea Verlinder rookie refractor of auto Lopez blue J's End of May Raleigh beam team rookie refractor CJ Abrams Washington been intended Crawford last pack box one here Trophy hunter Salvador Perez Yeah, Murphy and dance beat Swanson. All right solid first box and we got two autos out of that one Christian's dropping y'all the link for the next break again. We got the second half of this going up next Still six teams to go again our remaining teams are on sale and under a 50 bucks Team the future refractor Vladdy Thomas Get that one filled up and then after that we've got random division on an eight box full case break Raffy Deversby team of Bowman's best 32 autographs coming out of that one. I got trophy hunter Mookie Betts. Julio Rodriguez rookie base Random division style Handful teams in that break Carter team of the future Tati If anyone's been watching the dynasty baseball breaks that ripped yesterday What was y'all's favorite card that you've seen pulled so far got Corey Seeger 164 of 199 Rangers on the blue color match there Chris Taylor Donovan There was one. I can't remember what player it was. I'll have to look back at the pictures Buster Posey It was like a it said the patch on it was like a patch from some special event It said like Japan on it. It was a really really nasty Patch, and I cannot remember who what player it was off top of my head Don Mattingly on the on the green You know Yankees seven of 99 some of the craziest patches come out of that stuff Yeah, that's probably my second favorite. Yeah, that was out of was it a rod. Yeah, that was sick Really cool Freddy Freeman beam team Kyle pulled Bobby Whit. I think that one was like out of five. It was pretty low number that went That he pulled show.  Tim's got more of that going tonight They're already sold out spots coming in for the ones that were just listed to Sammy Long working refractor We've got wave refractor team of the future wander Franco. You know race Strawberry yeah, a rod such cool names in there, too Kirby team of the future will Smith refractor Man beam team Joey vato Looking for a lot of here Hopefully they get shorter one with that Fox have it too. Sometimes that doesn't happen. I like this photo Feel the dreams. That's cool. Jose Miranda rookie refractor All right, come on last pack Dynasty happy bias. We've got rookie refractor auto Nick Allen for the ace. There you go Oakland's Always makes me nervous and get to the last pack and no auto yet Chapman and Mackenzie Gore rookie All right, here we go box three. Let's see where we are on this next break y'all Yeah, we got the second half here. Just three to go. Awesome. Thank you Rockies Marlins and A's Rockies 15 Marlins 19 A's 15. So all three of those teams left under 20 bucks. Very nice Yeah Yeah, I figured for us. Hopefully it's just a bonus though. You can always you can always help, right? Again if you're not a fan of stadium club, we do have a eight box case break of Bowman's best That's gonna be random division style. That's the only other baseball going today besides stadium club for the daytime That is This was another one of my favorites friend off So definitely be sure to check that out get a handful of teams there for division After baseball, we've got a few F1 breaks to Rizzo beam team Which is nice some F1 sapphire. We haven't seen that in a while We've been out of that for for quite a bit, and we got a lot of F1 Fans beam team rookie refractor say a Suzuki Covey There is the link for you again last three teams all under 20 bucks Trophy hunter trait Turner And again more group breaks tonight with Sam certain at five and then Kyle will also be on drip again tonight We got a torque team of the future rookie refractor National hockey card day and nice. We have rookie refractor auto Jake Meyers Go astros another hit here Jake Meyers Uh-oh forces in chat. He heard John that one Patrick National hockey Bobby with junior on the destiny rookie refractor We are killer Brits. I got it. I got to make it to a lightning game one of these Seasons would be fun Platis beam team have never been to a hockey game before have you With my parents first date penguins Yeah, Akunya junior. We've got a wave refractor trophy hunter of Vladdy Ray and happy bias. Oh Man You can I'm sure you could go to the local target. There's I'm sure there's some hockey sitting on the on the shelf for us Robinson refractor Yeah There's a lot of everything honestly Uh trophy hunter refractor Pujos. It was funny. I ordered a couple of prison football blasters off target And I was just sitting in the office yesterday with the baby. I'm like I feel like ripping a box So I let her pick which box a hundred green blue rookie 133 a 199 for the Reds It's like alright Marley, which box should I rip? She like grabs for one Think I was gonna bring the good luck. No my best car was like a green rookie of Chaco from the from the chiefs team of the future rookie refractor shambas. I was like, all right Thanks, Marley Go to your local. Let's see. There you go. I got some got some hockey in the front All right, that'll do it for box three on the box for Solar bear games. Yeah, that's true and it's a lot a lot closer than Tampa, but I'm sure the the Tampa games are like 10 times better for sure refractor aeronado my friend from college actually works for We've got orange nice that cranky for the Royals that's gonna be 12 out of 25 First orange of the day and behind it. Wow, that's a sick photo there Gavin sheets on the rookie refractor white socks That's a cool one one of the last teams taken Grats on that Chicago Razz Pizza man, just Rockies and A's left now y'all down to two Rockies and A's believe those were both $15 teams half case break not bad Kyle Tucker team of the future burger Minorly cocky Usually We've got blue Jared Kelnick for the Mariners that one is 79 of 199 No such thing as cheap and drinks when it comes to the Amway Through there last night paid like $12 for a Miller light beam team Acunia Kind of ridiculous. I remember once we went to a magic game And we've got green trophy hunter Acunia a nice one there for the Braves 50 of 99 We got two drinks. It was like whiskey and coke or something. We got doubles $50 Yeah, yeah, it was it was kind of ridiculous Romero rookie refractor But all right sticking to the beer See with the future refractor rookie wander Franco What do we sure did Harold? We sure did Casey's on central. Yeah We've got green Nolan Aronato. I want is 12 out of 99 By the time we got there and all the traffic on I4 as I'm sure you know just We we left like an hour and a half early, but there is a crash So of course it was all backed up and got there barely before game time. So we didn't have any time to creating dynasty refractor pools would Do trophy hunter Joey Vano? Clark and Hicks Like we already ordered drinks too. So it's not like you can say oh 50 bucks No, we don't want them anymore by Frito on the rookie refractor Team of the future one Soto now, you know going forward No doubles Just beer. All right box five here Uh-oh what force do What it for us to do to get banned from a bar? Let's see who we got. It's probably trying to sell sell break spots. Hey, man. Yeah, just the Rockies left. You should go buy him Rockies 15 bucks up next Only team left down to one rookie refractor Kyle more Soliciting at the bar beam team Shohei Otani refractor a trophy hunter Bryce Harper Tuve Castro The next F1 race. He said it was next weekend, right? The 28 Where they reached in this time, you know where Kyle Lewis another numbered card here for the Mariners 31 of 199 Seattle Oh fellow of a barstool Team of the future Sean Murphy to be That sounds painful Got these destiny. We've got orange rookie Otto for the rays of Tommy Romero Nice orange Go Champa Bay five out of 25 there. Nice Piazza They haven't been Champa Bay in a minute shirts or trophy hunter burger rookie Beam team rookie refractor Torkelson One swing Gilbert Wall Street is our go-to See in the future rookie hunter green Wall Street so he's a bad Especially like after like weekend Magic Games or like Orlando City Team of the future Austin Riley We've got gold Tommy Edmund for the Cardinals 21 of 50 there auto Lopez Ricky Paul Muller on the refractor and we've got Marlins a Brian De La Cruz rookie refractor They go Jess when the last team's taken there what 19 bucks got yourself a nice little rookie Otto grats on that Grown Reynolds beam team rookie refractor Wanda Franco race Yeah That's definitely changed over the past few years. That's for sure box six here Awesome, we are sold out for stadium club So that means we are looking to fill our random division break still five left in Bowman baseball is a full box full case break eight boxes. I'm gonna get you the link here Five to go There is the link that tweeted out. All right Here we go box six Yeah, post-COVID Are sure beam team Bryce Harper Ashby everything post-COVID lowkey Mike trout trophy hunter refractor Schwerber Torque on a team of the future refractor Tigers Morales Larkin Only one baseball break with spots remaining for the daytime a Taylor Ward green for the Angels 80 of 99 And a green We've got blue Eddie Murray for the Orioles five of 199 there Olson Gray Geolito Yeah, downtown Sanford is fun too. We're like five minutes away from there now. Let's Close by where we moved Julia Rodriguez destiny rookie. Nice one there. There's beam team on the refractor wave torque. Nice rookie pack here Stanford brewery Cool place we went a couple weekends ago and they had like a whole block Closed off Raffy Devers trophy hunter. There was some some event going on Yeah, Harold for sure team of the future Chisholm So if you want to join five There's five spots left. So I'm doing one second. I got you 109 of 199. We've got friend Malraeus for the Cubbies Merrifield Will Clark and Ozzie Smith. So if you join that break, we'll do a random right before the break You'll get a sign a random division Wellness and Whatever T or cards we pull from that division, you'll get all of those cards set to you Shane boss team of the future And it's gonna be an eight box full case break of Bowman's Best Baseball So that break you get a whole division to yourself Winter garden area. Yeah, they're really gone to winter garden too much. They're G refractor is a cool one for the Yankees and rookie refractor auto, it's gonna be a Simon Muziati for the Phillies Phillies doing well There you go And last We've got Aaron judge trophy hunter Ted Williams, Yelich and Badoop All right box seven correct. Yeah, it's random We use a random.org will type in all the names of the people who bought in as well as the divisions Randomize the names randomize the divisions put them side by side and line them up and that's what division you will get Yeah, so it is totally random. I've got green Charlie Blackman for the Rockies that is two of 99 and Dodgers rookie refractor auto is gonna be a Andre Jackson Nice autograph there Andre Jackson you shadow There is a link for you again five spots to go on that It's gonna be the last break a baseball break of the daytime. I like Manoa the only one with spots remaining Then last one before we get into some F1 on refractor Penny and Ricky base Andrew Jones See a red got refractor to grown Walker Kellenick Let's see one beam team rookie refractor O'Neill Cruz Like you can feel quickly See that slide in the home, but I heard it was nasty. I can't watch like sports injuries. I just can't Tear my ACL four times three out of the times being some basketball. It's like no Turn my head away might shout on the dynasty refractor Stromsky team of the future Alonso Again, y'all hope everyone's having a fabulous start to their Saturday morning We've got beam team Alonso appreciate you tuning into the stream buying a spot popping insane. Hi Sam will be live tonight at five for more group breaks Rizzo as well as Kyle over on drip or from some person else Kyle's been on it on a heater man Let me tell you the bolts are really nice guards over on drip beam team refractor aeronaut Trophy hunter Greg Maddox sprays Harper beam team. That is a gold beam team nice one there 30 of 50 sneaky tops try to be sneaky there Nice one there again Billy's doing very well Chris Bryant George Britt Torque on the rookie base. We've got blue Matt Olson for the brave zone is 121 of 199 Lou and Hank Aaron Again, all the other pick your team breaks also are on sale. So definitely be sure to check that out team of the future Acunya That goes for prism basketball pick your team and then the new release as of yesterday a playbook football Those will be the last couple breaks of the day We got four box breaks of prison basketball I got three of those pick your team style and then half case breaks, which are eight boxes of the playoff football beam team rookie refractor Julio Rodriguez Cormac Hayes Nella like a Florida rip in some playbook. Oh, he's a cool product trophy hunter Tati's Well, we've got refractor Salvador Perez I Team of the future refractor rookie Bobby Whit jr. Yes, sir. You know, you've got orange Matt Chapman for the blue jays 13 of 25 Couple oranges coming up our beam team with the bets second half here got trophy hunter Juan Soto Winder, you know Cruz with the base Fractor Geryl Jr Tina Let's get this bullman's best baseball filled up again random division style on that one and we've got rays again rookie refractor auto of by Dal Bruhan on rookie refractor Oh time and her shop Blue Justin Turner nice doctors color match there. That one is 151 of 199 Junior She's um junior on the destiny. We've got a destiny wave refractor wander Gray's also doing a couple autos some nice wandering Few packs left here Beam team rookie refractor Bobby Whit jr Josh Lowe Vader trophy hunter Carlos Carrera And last pack here and then we'll do a quick recap a lot of him and as Gavin sheets rookie All right, let's go ahead and do our Recap again, we got orange is going to be out of 25. We had Matt Chapman for the blue jays Is that granky royals? Vidal Bruhan rays Andre Jackson Dodgers Simon Muziati Phillies Brian Dela Cruz Marlins Gavin sheets white socks Jake Myers Astros Nick Allen a's the Jose Miranda twins Bryson Stott greens and 99 for the Phillies and orange rookie auto Tommy Romero for the race that will do it for the first half on number seven We've got the second half number eight coming right up. Thanks everyone


The River State Governor Nisamuike says the recovery and eventual return of the seemingly lost asset of the state, Legacy 600 aircraft, is a reality and not mere politics. Governor Nisamuike spoke after receiving the Legacy 600 aircraft that touched down on at the Port Harcourt International Airport on Moa in the Quarry local government area of the state. Governor Nisamuike explained that he took intelligence report for his administration to discover such asset owned by the River State government. In 2008, 2019, that look, this asset is somewhere, I'm going to sell it, I was there, which asset is that? The Legacy 600. I said no, it's not possible, we have to send people to go and investigate and they found out that that asset is there. It's all and all to the glory of God, the plane is back. And if as people can see, Nigerians can see, it's not that we are playing politics. All you have to say, look, we never knew. Nobody told us on to God intelligence that we have this asset somewhere. If we had known on time in 2015, the kind of money we spent wouldn't have spent another month of money. Don't forget to hit the notification button so you get notified about fresh news updates.


Hi Crystal, this is Ashley with Oxmo Toyota. I wanted to send you over a quick video greeting that way you can see who you'll be working with. I noticed through Truecar US News that you were interested in a new 2016 Camry. I want to hit and send you over a little bit more information about our dealership. When you get a moment, please give me a call. My number is<PHONE_NUMBER> where you can also email me back. Thanks. Bye.


Hello everyone. Thank you all for coming. My name is Shalini Shekhar Munger. I work at Lucidworks And I work almost full-time on Apache solar focusing specifically on solar cloud and Finding its bugs hardening it testing it to its limits, you know and doing that over and over again So in this talk, we're gonna talk about Jebson and flaky networks. What is Jebson? I'll come to that in a bit. Sorry about that Let's start with What do I mean by a flaky network? So this is from Jeff Dean who works at Google at LADDIS 2009 Admittedly the stats are a little bit old, but the message is still true today In their experience in the first year of a new cluster a cluster has about between 40 to 80 machines about five racks have problems in which There's a packet loss of almost 50% so 50% of servers can't randomly not be able to talk to each other and You have network maintenance events in which you have sporadic connectivity issues between servers router failures DNS issues what not right and this is Google these guys know a few things about running large clusters, right? so the fact is that Reliable networks are a myth. They don't exist when you have asynchronous communication happening in a data center things will go bad and It's not just the network. It's not just the network equipment which goes faulty It's also your systems. For example, if you're running Java You have garbage collection pauses, right? So processes which will fail to send a message or respond to a message because they are in a garbage collection pause and That is not any different from a network partition if you're looking from the outside, right? If I'm trying to send initiative a request to a server, which is in a garbage collection pause It's not any different than if there was a bad firewall rule which prevented the packets from reaching that server There is no difference between the two, right? So network partitions are a reality It's not just the network equipment which goes bad. It is process crashes It is the scheduling delays of the Linux kernel For example, you're running your systems in a cloud on AWS or other shared hardware, right? With virtualization and if the other virtual machines on that same physical host are doing are very busy and the the virtual The the scheduler decides to give more priority to some other virtual machine Then the same thing happens. Your processes cannot run They'll be paused until the the scheduler comes back and gives them priority right So when we talk about network partitions, most people think that the network partition happens something like this So this is a five-note cluster So I've named the machines n1 n2 up to n2 for n5 and this dotted line in the middle can be a bad firewall rule It can be a faulty switch which it refuses to send packets across It can even be a garbage collection pause happening on n4 and n5 making Keeping them completely separate from talking to the other nodes in the cluster Right, but this is not the whole truth. This is not the only way that network partitions happen In an asynchronous network, it is allowed that packets can be lost. It can they can be completely dropped They can be duplicated. They can be reordered and they can be delayed Right and if the delay is too much, what do you get in your applications? You have timeouts. You have connection timeouts. You have read timeouts, right? So all this are all these things are allowed in asynchronous networks right So when you have these kind of things, how do you? How do you figure out what a distributed system is supposed to do in the presence of these kind of problems? So let's come back to a topic which has been discussed at least twice yesterday in different talks the cap theorem The cap theorem says that you have consistency availability and partition tolerance and you can choose only two Right, and this has been discussed to death yesterday and in literature and you know in different research papers as well So what is consistency according to cap theorem? Actually, the cap theorem is very very specific about the kind of consistency it requires it requires linearizable consistency What that means is from an external client all operations that are happening in a system are supposed to happen They should look like they are happening at the same instant Okay, even though they are not happening exactly the same instant in the presence of concurrent Read and update requests, but from an outside perspective. They should look like they are happening at the same instant, okay? Availability basically means that every request which is received by a non-failing note Should respond it does not say how soon should it respond? There is no time bound on it, but it does say that there is a finite time bound It's not like you can just give up and not respond at all So you should respond at some point, but it does not say how soon it you should respond and Then you have partition tolerance what that means is during a partition an arbitrary number of messages may be dropped It only talks about dropping messages. It does not talk about Reordering messages or delaying messages, whatever well the delay is actually quite similar to a drop But let's not go there for a moment So partition tolerance says that unless the complete network is shut down There should never be a incorrect response coming back from a machine But when it comes to real systems that we use real data stores and databases that we use like Cassandra Elastic solar mongo, RIDIS, etc. etc How does that manifest practically to your applications? when you choose to forego availability so Partition tolerance is a must right you cannot not choose part in tolerance a system should not I mean if you Partitions are going to happen no matter what you do no matter how reliable your systems are right So if you you must choose to tolerate partitions So which means you can either give up consistency or you can give up availability for different operations And it's not a it's not a hard and fast tool that you have to do it for all kinds of operations So different systems can choose different kinds of guarantees for different kinds of app for different kinds of operations But when you choose to forego availability Then what happens either the server or your data store responds with an error saying, you know I'm not available right now. Maybe try again later or you might not find a route to that host So you get connection timeouts or maybe that host is actually trying to make a call to a resource on some other node In which case your request can be hung for a long time and you can have timeouts in your applications At the same time if you if your data store foregoes Consistency then what happens you can get stale results So basically results which are out of date which have been overwritten by another process on some other part of the data Store on some other machine or you can have dirty reads.  For example, a system might choose to Give you give the client a new Response a new data even though that new data has not been acknowledged back to the original guy who wrote that data So that is a dirty read so some systems can do that the third thing is and the most catastrophic thing is it loses your data The data which you added which you updated to the system and you got back an acknowledgement from that system is now No longer there at all. It's just gone Right, so this is like the most catastrophic kind of a kind of a failure when you forego consistency How do you plan for these things? To plan for these things you first have to figure out whether your data store actually exhibits these behaviors and which Behaviors does it exhibit now a lot of the literature a lot of the documentation for different data stores? Sometimes it's not complete. Sometimes it's a little hand wavy, you know, not very precise or sometimes it's not present at all and The only way you can actually know the guarantees of your systems is by testing them so how do you test for these kind of problems and That is where Jepsen comes in So Jepsen is a tool written by a guy called Kyle Kingsbury who goes by the handle of if you're online and He has written Prolifically about different data stores. He wrote this tool in closure What this tool does is it tries to set up a small five-note cluster and Then it tries to wreck havoc on that cluster by either using IP tables to drop packets or to delay packets Or to SSH into those individual boxes and crashing the processes without giving any warnings or maybe pausing and restarting those processes Or maybe it can even do things like go in and change the clock of the system or go in or Flip the bits on the disk for the different data source, right? So it allows you to create very very powerful tests, which you can use then to test on your own systems So what does a typical Jepsen test look like? A typical Jepsen test has these components First of all, you have a bit of code to set up that data store This data store can be Cassandra, Mongo, whatever whatever you want to test, right? So you will set up this in an automated fashion and you can tear it down So what we typically do is For example, how I test it is I have this one large desktop machine with the eight cores and 32 gigs of RAM Well, it's not that big now, but it was big with four years old four years back when I built it and Inside that I have Linux containers running So I have five nodes running as different Linux containers in each of these five nodes I set up solar and I have these clients which are running as part of the Jepsen process on the host machine Right and this host machine has permissions to go and SSH into each of these Linux containers So that we can do these kind of operations So you have the automated database setup You have the client or the test definition and we are going to talk about that in a bit You have something called a nemesis, which is the guy who creates chaos It's like a chaos monkey kind of a thing, right? So it creates chaos in your cluster Then you have a schedule of operations So you can say, you know, I'm gonna write a new data or a new document every one second Then I'm gonna wait for a random amount of time and maybe after every 30 seconds I'm gonna create a partition between nodes and then heal that partition after another minute or so Okay, and I'm gonna do that repeatedly for maybe five minutes or 10 minutes or whatever. Okay? So you have the schedule of operations Whenever you whenever the client makes a request whenever the client invokes an operation on the host Which is running on the Linux container You either get back an okay response a success or a failure or it times out which is basically ambiguous It could either have succeeded and you did not know about it or it could have failed So you have to treat that as an ambiguous response. It can either be a success or a failure message So all these success failure and indeterminate responses are logged Which we call as the history of operations and this history of operations is important because that lets you figure out What should have been the final state of the system under the guarantees that are provided by that data store Okay, and then you have a consistency checker which basically figures out whether the final read was consistent Or whether it lost data or whether there is some extra garbage, which we never actually put in in the first place Okay, so the first three things are data store specific So if you are testing mongo, you will write drivers a client for mongo You'll write the automated setup code for mongo and and the nemesis types Some of the nemesis are actually provided by gypsum and we're gonna talk about that in a bit But this part is data store specific So if you're gonna test something of your own you're gonna write this part in closure according to your own data store and Then these three things are provided by gypsum. Okay, so you have these three nodes Which are created using Linux containers. You have these three clients Which are created inside the gypsum process They make request concurrently to these different nodes and they get responses, which can be okay or failure or indeterminate and Collectively these responses are known as the history of the system Okay Now what is a nemesis So the dictionary describes the nemesis as the inescapable agent of someone's downfall and this inescapable agent of Someone's downfall that someone is your data store. It could be mongo or solar or whatever, right? So let's look at a few of those nemesis that are provided by gypsum for you So the first one is pretty simple. It's called the partition random node It will randomly choose a node from your existing cluster and it will SSH into that box and use IP tables to drop all external traffic So both incoming and outgoing so that node will neither receive any traffic Not it will be able to send any request out of it, right? The second one is very similar But instead of using IP tables it will just go inside that node and do a kill minus nine It will just kill it without any Warning at all right and then it will do all these tests the third one would be it will go and change the clock It will change the clock Maybe make it five minutes before five minutes afterwards or whatever you can actually specify the threshold of how much the clock is supposed to change Then you can get into more complicated tests where you partition halves so if you have five nodes you will have a majority partition and a minority partition, right and You can choose them linearly so you can always say that hey N1 and N2 and N3 are gonna be together and N4 and 5 are gonna be in a separate partition Or you can choose them randomly, which is the second one here the partition random house The third one is even more interesting. The third one is called a bridge partition In this bridge partition what happens is the node N3 can talk to the nodes on both sides of the partition But the nodes N1 and N2 cannot talk to N4 N5 and vice versa, right? So this is a very very tricky sort of a partition and it's not like that this partition doesn't happen This partition can also happen in physical and real networks So how do we write the client? How do we test? What are we actually gonna test? So this is a simple test that I used for solar It was originally written for elastic by the guy who wrote Jepson and we adopted the same test to test solar else as well So in this case what we're gonna do is we're gonna create one document Inside a data store which is going to keep a list of integers a set of integers Okay, so those integers should not repeat and We are going to use the compare and set primitives available in these data stores to add a new integer to this set Right, so we read the data store. We see that the current version is V1 Then we say, you know add an integer 2 to this set as long as the version is V1 And then you can keep on repeating from each client Okay, so presumably only one of those compare and set operations should succeed and whatever has succeeded should never be rolled back right So we do this concurrently from each client So if you're going to set up a five node cluster to test We are going to create five clients as well and we are going to ask them to use multiple threads to write New integers to this set stored in that data store okay, and Then we'll create and restore partitions in the in the middle and at when finally the all the partitions have healed We'll go and read the set of values which are stored inside the data store and see if they actually match What should have happened? Okay So in this case, this is a visual representation of the same thing You initially have an empty set and you do a compare and set and add the value one to the set of integers So you have the value one inside stored as a document inside solar or elastic or mongo or whatever Then you have two clients which are trying to update Which are trying to add a new value to this set concurrently So because they're trying to use the same version one of them will succeed one of them will fail Right, and then we'll repeat this process again and again During this time we will create a partition and then heal the partition and we'll keep doing this a couple of times to see what happens so This part whether the operation succeeded or failed or was indeterminate is the history of operations The history is nothing But the time at which the operation responded back to you the operation that was actually being performed and the result of that operation so originally the author of Jepsen He Wrote a series of blog posts which are really good which you should all go and read about if you're interested in distributed systems and He done he did tests on redis on mongo on elastic zookeeper Cassandra aero spike Redis dis queue a lot of different data stores He did not test solar But when these results came out a lot of our customers a lot of people started asking That how well does solar perform in the face of these kind of partitions? so since it is one of my Responsibilities in my job to do these kind of things So I started learning closure and I started writing these tests adapting them from the ones that he had written For elastic and running them on solar. So this is a process. It took about two to three months for me and we found a few bugs and We did solve those bugs So this is I'm gonna talk a little bit about what we found and how does it actually perform? And then I'm gonna come to other data stores So a solar in how many of you are familiar with solar? Okay, a lot of you are familiar with solar. So a quick recap solar is a Search server which uses lucine are not very different from elastic in that sense But there are a few differences For example solar uses zookeeper for distribute consensus. So it relies on a zookeeper cluster to figure out Which note should become a leader or who is the current leader or whether a particular node is live or dead and so on and so forth It has a lucine index. So you can add new Documents you can do a full-text search on those documents You can do autocomplete highlighting faceting analytics so on and so forth It also has things like compare in set So you can atomically update a given document by providing the last version that you know about and if the version is still the same Solar will go ahead and update the document. Otherwise, you it will say, you know, there's a conflict and I cannot update So presumably because it provides a compare in set and an atomic primitive Presumably it should be linearizable. Okay, so let's see what happens When you add a new document to solar, it is synchronously replicated to all the nodes which are available at that time So which nodes are live at that time is determined by using zookeeper. Okay If you want to read the whole thing like the complete test methodology and the results and in the configuration everything There's a blog post I wrote last year in December. It has all the juicy details You can go and read it In this test, I'm going to use a compare in set client I'm going to simulate a set of integers going to add an integer every second concurrently from five different clients on five different solar nodes and I'm going to partition the network Randomly in half every 30 seconds and I'm going to keep running the partition for 200 seconds So for 200 seconds nodes in the in the majority in minority partition will not be able to talk to each other Okay, I know you cannot read this but it's not that important Just closure. I wanted to show I can write closure So are we actually safe the solar actually do what we wanted to do? Well, uh, maybe The thing is uh, every application is different if you are going to use solar as the only data store Then you're going to have some problems But we do not have any catastrophic data loss problems which were found in other data stores What we did find was that when a partition happens the leaders on the minority partition They become unavailable For up to the zookeeper session timeout, which is typically 30 seconds in solar by default So for 30 seconds until the time that solar realizes that the leaders are no longer Reachable from a majority partition. It will not elect a new leader But after 30 seconds when the zk session timeout happens It will elect a new leader on the majority partition and then things will continue as they were before So for that 30 second or maybe a little bit more or less than that solar writes on solar For the leaders, which were on the minority partition will be unavailable So you can figure out whether that is uh reasonable for your application or not So there is no hard and fast rule on these things Some rights will hang for a long time And of course timeouts are essential Uh, so when I say for a long time, they actually hang on for some of these rights Which happened right as the partition was happening would just go and hang for as long as the partition Was kept up Which is pretty bad But it's not so bad because in in in your real practical applications You will set up a timeout for maybe three or four seconds or something like that And after that you will say hey, the operation did not succeed and maybe I'll try again later So not that a big deal Under the comparison operations and under these kind of partitions The final reads had all the integers for which we got an okay response Plus some extra integers for which we were not able to figure out For example, the the request timed out So we were not able to figure out whether they failed or whether they succeeded And some of those actually succeeded and they were also found in the final set So on the right hand side, you have a small snippet of the results which are generated by Jepsen Which says the recovered fraction and this is a ratio So which means for 14 requests the results were ambiguous and they were actually found in the final results So the 14 were recovered Zero were unexpected. So there was no dirty reads There was no nothing which was which should not have been there which we never added in the first place You we did not lose any documents And the okay fraction was one to five one out of two three five nine total attempts So when it comes to availability, well, probably not that great But then this also depends a lot on the cluster topology that you're testing Right For example, if you're running just one shard and five replicas The availability results might be very different than if you're running three shards and five replicas or five shards and three replicas Right, it all depends on where the leaders of those Shards land up and whether they land up in the minority partition or not So no data loss yet, which is good But uh, we have not proved a linearizable history For example, we did not look at the intermediate results and figured out whether the intermediate history was also correct or not Was there a total order or not? We have not proved that yet. So we'll do that in the later iterations of these tests But Just because we did not find very bad bugs that doesn't mean they don't exist Right, so we're gonna keep looking. So this is now a process Every time we have a new release coming up in solar, we run these tests figure out whether it failed or not Go deep investigate what happened. What really happened? Is that a valid bug or not? What happened? You know, and we just keep that doing keep doing that all the time for each release So that's solar What were the bugs that were found? so I'm not going to cover this very deeply in this because you can go and read up on these bugs online if you just search for these things One very bad bug that we found was that if you are issuing commits on solar during a partition Uh, it would put randomly it would just put any node in the downstate And then that node will not receive any traffic from the client and that's pretty bad What was happening was that all the rights that are done in solar are coordinated via the leader Right, but the commits are just broadcasted by any node because you don't need to be a leader to broadcast a commit It does not change the actual data. Which is inside the solar, right? It just makes it visible So what was happening was that the node which is broadcasting the commit was also acting as a leader for that operation during that bit of time and therefore it was able to Put another node another node in a bad state when it could not make a request that node, right? Which was pretty bad. So we fixed that now the commit operations don't go through that code path anymore The second one was not catastrophic, but it was a bit of an annoyance really what was happening was when The nodes reconnected to zk It was invoking the startup logic which is to recover from the transaction log and then Become available and that was not a bad thing. I mean that was not bad in the way that Anything any operation was being repeated because all of these operations are stored with their versions in the transaction log So if you're going to repeat them, it does not change anything at all Because the versions are old solar will just discard them But it takes time to go through that transaction log which can be hundreds of megabytes and actually invoke all those operations and then figure out Hey, this was all users in the first place Then there was some other I I already talked about the fact that some requests hung around for a long time Which we fixed the cluster status API of solar was a pretty was a little flaky It used to go to zookeeper each and every time Which gives you very consistent responses, but during partitions it tends to hang So it would instead of saying that you know, I cannot go to zookeeper It will just hang for a couple of minutes and not give you a response at all which is bad So we fixed that in 5.2 The last one was especially tricky to figure out this was a race condition between how a leader marks a replica is down and The exact time at which the partition happens and so on So the more juicy details are available on the JIRA issues. If you're interested in that, you can go and look it up Elastic so elastic is quite similar to solar, right? It also uses a lucine index and a transaction log. It's a full text server not unlike solar Promises data safety and consistent single doc reads and compare and set operations The search is of course like solar. It's eventually consistent For example, when you commit an operation, that's when the results of the ad or update become visible to searches So this should also expose this should also Show similar behavior as what solar Exposed, right? But unfortunately, that's not the case elastic uses a homegrown algorithm that what what they call as zen disco And it's not based on any uh pure reviewed Consensus algorithm like pack source or raw after whatever. I don't know what it actually does really. I've never looked at the code but what happens really is when And this test is not something that iron myself. This is a test that The guy who wrote jeffson if he did did these tests and he wrote a nice blog post summarizing them For version 1.1 that was initially when he wrote the tests and then again for version 1.5 And in both the cases he found a lot of data loss issues during different kinds of partitions And not just during network partitions, but also during garbage collection pauses so, uh They're getting better between versions, but they're still not quite there They still have little work to do. I think if they just use something like zookeeper They would avoid a lot of these problems But they choose not to do that to make it easy to set up probably and that bites you in production Okay Let's come to mongo. How many of you use mongo? Okay, and how many of you know about mongo like generally Almost all of you right? Okay, so mongo is a different right? It's supposed to be like a proper data store Not a search server like elastic or solar And it's a documented or document oriented data store Which means you have a document with a list of key value pairs The values can be further json documents and so on and so forth And you can nest them and it provides atomic updates on the documents as well So it is supposed to be like a An alternative to a proper database like mysql or postgres or whatever right? So in theory it should give you some of the same guarantees that those systems give you or maybe a little less Right, but what exactly are the guarantees that it gives? So first of all mongo does not use anything like suki power or anything It has uh, basically you have concept of replica sets Who elect a primary? And the others become secondary to that primary All the right operations go to the primary and the operation log is replicated Asynchronously to all the secondaries in that replica set And the replicas decide among themselves when the current leader is to be demoted or when One replica among them is to be promoted as the primary Okay And what i'm going to talk about here is applies to versions 2.4.3 and 2.6.7 So the way it figures out whether nodes are up or down Is very simple it every node sends a heartbeat message every 10 seconds And if a node does not receive heartbeat messages from other node for up to 10 seconds It determines that you know that node is falling down and if it was the leader I'm going to demote it and I'm going to promote myself or somebody else and so on and so forth So it's very simple in that manner So it claims to have atomic rights for a document and it supports fully consistent reads What is a fully consistent read so that is again Uh a bit ambiguous. It's not actually fully consistent In like when you start reading the documentation, they first say it's like fully consistent But then when you come down later in the documentation, it says that Uh, if you want strict consistency, you should always read from the primary because the the Updates are replicated asynchronously to the to the secondaries, right? If you're reading the secondary, you might get some steel data which has already been changed on the primary Okay, can you guys still hear me? Okay, so presumably if we write to Uh, if we always read from the primary, then we should be safe Right, but that's not the complete story There is something called A right concern in mongo, uh, which has different levels So when you're writing a document to mongo, you can specify one of or more one or more of these right concerns And it will do things accordingly in 2.4.3 mongo db used simple udp messages to send Data one way to the server and just forget about it It would not even know whether the server received that request at all or whether it acknowledged it So it was like completely unsafe. So they changed it after if it published Uh, a dog a blog post on mongo db and now they use something called an acknowledged right concern Which means that the primary acknowledges to the client that yes, I have written this data But this data is only written memory. It's not actually journaled to the top log or written to its actual file formats So it's not on the disc just in memory Which also means that if the primary goes down Before the data can be replicated to the secondaries your document might be lost Right, which is bad So you have right concern journaled fsync replicas acknowledged and then something called majority Which means it's going to acknowledge that the right actually has been sent and acknowledged by the majority of the replicas So this is supposed to be like the safe setting Right, so this should be perfectly safe But is that true? No, not really what happens is That when you're writing to a majority So first of all if you're not even writing the majority then when a new primary is elected in a majority in a new partition Then the data written to the old primary Will get rolled back eventually when the partition is healed Right, so you will have rollback of data for which you got an acknowledgement from mongo, which is pretty bad But there's a silver lining the rollback is limited to 300 megabytes only So only 300 megabytes worth of data will be rolled back if it tries to roll back more than that Then mongo will actually error out and say that node will go down and say, you know You gotta do some manual recovery steps. I cannot roll back more than this amount of data So safe, right safe So even when there is a majority right concern enabled When it's fsync and journaled and all those things Still you can have inconsistent rates and that is because the isolation level in mongo is read uncommitted Which means that if client a sends a new document to mongo and client b Sends a new document to mongo So let's say client a added an integer one to a document written in mongo, right So it will mongo will write the primary will write to its own op log in its own index And then it will try to replicate them to all the replicas and wait until the acknowledgement happens But in the middle of that if another client comes in and tries to read the value From the primary it will give you the value, which has still not been acknowledged To the actual client who wrote the value, right? So what does this mean? So this means that let's say You have a service where you allow users to change their username Uh, or the display name something like twitter, maybe right So what happens is I change in my name to x and you also try to change the name to x Which should not be allowed, right? But the primary that I wrote to Went into a temporary partition. It was in the minority, right? So eventually this primary will give up its primary ownership because it is now in a part in a in a partition And the other one will assume The primary owner of that shard the one on the other majority partition But in the meantime Somebody comes, for example, you change your name and then you attempt to see the Uh preferences of your account Somebody will come and try to do a read from the old primary. He will see the new data And if it was you who was trying to see your preference pages Mongo might show my data to you Which is very very bad. So in anything which requires some sort of a linearizable constraints Should not be done on mongo at all. It's just not safe under any right concern under any sort of replication strategy So What do we Take out from this? What do we learn from this? Uh session the first thing that we learn is network communication is flaky If you have asynchronous communication things are going to get dropped.  So it's better that you find that surprise yourself first before your users find that Out for you Right. So jebson is what you can use to test for these kind of problems And if you want to learn jebson, if you want to if you already know closure or if you want to learn closure and Do some more tests We can always use some more help testing solar So help us find some bugs so that I can fix them And that's pretty much what I wanted to cover. So, uh, before I Close this I just want to say one thing that we are going to have a solar lucine meetup Next week on saturday at the target office in manatek tech park. So if you guys are around and if you're free Come by we're going to have a few good sessions And now I open up the deck for questions Thank you For solar, I mean through jebson. Would you also put zoo zookeeper as well under jebson or yes? Yes, I do. So the so we tested two different topologies in the so In the first one, we had zookeeper running on the host machine Which means that it is never partitioned away In the second topology, we had five zookeeper nodes co-located with five solar nodes running on each of the Linux containers. So whenever there was a partition the zookeeper also partitioned into majority minority So zookeeper actually performs pretty well. Uh, if it himself tested zookeeper and he gave it a clean Bill of health a good, okay So zookeeper, uh, compare and states are linearizable Of course, it has some because it's a consistent data store It does have some problems with availability But that is to be expected and actually it's a good thing for zookeeper to have those problems So have you tested elastic serves performance In case when it undergoes, uh, bridge partition Yes, actually, uh, I think one of these, uh Okay, so there is a bridge partition right the second bullet here This is these are the results in case of the bridge partition So if it tested it against random node partitions under partition halves portion random halves and the bridge partition as well as he simulated a test where He tried to simulate a garbage collection pause So he would go into the box and do a six stop and say continue So it will pause the process and then unfreeze it again to simulate a garbage collection pause And it had these data loss issues in almost all of these partitions. So it's getting better It's definitely getting a lot better But like for example, it had like a 29% loss in the in 1.1 and only a 3% loss in the second one, but You know Whatever is acceptable to you


Imagine for a second you're sitting at church and all of a sudden the pastor up at front decides to get everyone to recite the Sparkle Creed. I invite you to rise in body or spirit and let us confess our faith today in the words of the Sparkle Creed. This is pretty insane and we need to know as Christians how to respond to this type of lunacy. This video is brought to you by my patrons on Patreon. This is my full-time gig and it's you guys that support me and what I'm doing to equip people to follow Jesus daily. So thank you so much and if you want to support what I'm doing click the link in my description. I invite you to rise in body or spirit and let us confess our faith today in the words of the Sparkle Creed. I believe in the non-binary God whose pronouns are plural. I believe in Jesus Christ their child who wore a fabulous tunic and had two dads and saw everyone as a sibling child of God. I believe in the rainbow spirit who shatters our image of one white light and refracts it into a rainbow of gorgeous diversity. I believe in the church of everyday saints as numerous, creative and resilient as patches on the ace quilt whose feet are grounded in mud and whose eyes gaze at the stars in wonder. I believe in the calling to each of us that love is love is love so beloved let us love. I believe glorious God helped my unbelief, amen. Okay there were a lot of problems with that obviously but the first thing that comes to mind is that this church is trans and I don't say that to be hyperbolic but it is pretending to be something that it's not. This church is pretending to be a church. This pastor is pretending to be a pastor. This congregation is pretending to be a congregation of believers that are gathered around to worship God maybe. They're pretending to be something that they're not because they're worshiping an idol. Now the funny thing about the creed is that it says that God has plural pronouns and you look at a verse like let us make man in our image right in Genesis.  It's like no no no three distinct persons father son and holy spirit and yet one and so that's why it says let us make man in our image. It's affirming the doctrine of the trinity. That is not a they now when you watch this video it's plain to see that they are engaging in worship that's obvious but they're not engaging in worship of the true God they're engaging in worship of their own behavior their own sexuality their own beliefs their own ideology themselves putting their faith in themselves there's no faith being put in Jesus at all like when you see what they're noting in their creed right this idea that Jesus has this awesome tunic and he has two dads and it's like and God is they them pronouns that's not worship of the true God that's worship of their own ideology and you can see so clearly what they're worshiping and it's not God because there's no mention of Jesus stung on the cross just no mention mention of the gospel of his sinless life it's all about trying to pull things out of the Jesus story of the Bible that will affirm their own agenda that will affirm their own beliefs things that aren't even true or things that are phrased really weirdly like Jesus had two dads it's like that is so weird why would you do that when you do it because it affirms what you already believe about the world and your own preconceived notions about how this all should be oriented of course you know Jesus has two dads that's good like he's got two to gay dads it's like they know that's not true but they're still gonna say it why because they want to be provocateurs because they want the biblical narrative to affirm their beliefs there's a lot of reasons for it ultimately it's rebellion ultimately it's rebellion against God in worship of things that are not of him I think of Aaron when he made the golden calf Moses was up getting the Ten Commandments and everyone's like hey Aaron we want somebody to worship we want somebody to continue to lead us and so Aaron's like okay let's get all the gold together that's gonna you know melt it down and form this golden calf and it will continue to take us to where we need to be let's make a sacrifice to it and everyone starts worshipping it and you're like what is going on okay why are people doing this well number one God's commandments are absent okay Moses is on the mountain he's getting the commandments God's commandments are absent and everyone turns to idol worship you think about this sparkle creed church are God's commandments present well everyone's got their conscience right but they're in rebellion against him so there's no adherence to the commandments when it says you know love the Lord your God with all your heart with all your mind with all your soul and love your neighbor as yourself like the two greatest commandments when they when Jesus talks about the the whole law of God being summed up in those commandments is there any adherence to those things not at all those things have been disposed of for self love for what I believe I should be now we were all there at some point worshiping things that were not God rebelling against him but God and his grace saved us and that's what we're calling people to is to recognize these are idols these are not things that will satisfy you and in fact you were rebelling against the God who created you by worshipping them she uses this phrase love is love this is a really common phrase simply saying that you know all love is basically equal whether it's gay love straight love trans love whatever place on the spectrum polyamorous love it's all love but then you begin to ask people a little bit deeper and you say oh what about a man that loves a little boy in that way is that okay and maybe your culture is going in the direction where they begin to say yeah more and more yeah it's okay when you look at minor attracted persons or maps it's like okay this is absolutely insane that our culture is accepting this it's becoming more normalized but I think for most people they would still say no that's wrong that's a wrong kind of love and we kind of find a real important truth here that love when it's oriented at the wrong thing it's wrong right when love is oriented in a wrong way that's not love and so when we what we should do is we should love what is good and hate what is evil that's actually what Romans 12 9 says let us love what is genuine abhor what is evil hold fast to what is good if your love is oriented in the wrong way if it's directed at the wrong object that love is not good now just in case there was any confusion this lady is not a pastor this church is not a church it's just a building where people gather and they hang out and they say bunch of stuff and it goes viral on social media and it's it's a trans church a church that pretends to be a church but it is nothing of the sort what do we do with this outrage what do we do with this maybe holy anger that hopefully we have towards this just disgusting perversion of Jesus in the gospel because it's okay to have anger towards us that's a that's an alright thing but the Bible says be angry and do not sin and so we need to understand also where we stand before God that we on our own we have nothing we are only saved by the grace of God that we're just beggars that have found bread and so we shouldn't stand on our high horse but at the same time we do have a standard that we can say this is wrong this is this is just disgusting and a perversion and taking of things that are holy and bringing them down to the level of the muck and the mire so that's what we we can say to this absolutely but we shouldn't let that just stew in that anger or stew in that frustration at the world but rather let that lead us to movement to firm up our grasp on the gospel what is the gospel truly do you know the gospel because maybe you're watching this sparkle creed and you're saying this is ridiculous this is stupid you know this is a mockery of church and of the gospel and of Jesus and that's alright and that's good but if you don't know the truth if you are not able to verbalize okay what did Jesus actually come here to do right what was his mission here if it wasn't just to wear a fabulous tunic and have two dads then what was here to do because that's what's gonna change people's hearts it's going to be the gospel it's going to be the Holy Spirit working through us as we verbalize the truth to these folks that are so deceived and in rebellion against God so what did Jesus come to do well he came to live the sinless life that we couldn't live and die on the cross the death we deserve to die for our sins against God that's something this lady this pastor lady will never tell you about sin and about judgment and about wrath but it's only when we understand those things that we can truly understand the grace and the mercy and the redemption that Jesus has provided in his sacrifice on our behalf and the good news is that through that receiving of his grace and putting our faith in him repenting for our sins we find a new identity in him where we no longer worship our identity of who we once were of our sin of our rebellion of our sexuality whatever that looked like of a perversion but now we find a new identity in God and and who he calls us and he calls us his children and now we delight to follow him it's not this this heavy burden on our shoulders and how I need to be good I need to measure up to the standard of perfection but rather Jesus has met that standard of perfection and now as his child I can delightfully pursue the calling that he has on my life and this is an amazing invitation that we have to bring forward to people and so this is the distinction here don't let this outrage of what you're seeing in this video lead you to despair but rather let it lead you to delight knowing that Jesus is victorious over all of it and he is equipping you even now to go out and share this gospel with people that are deceived that are in rebellion and he's going to work through that and that is an amazing amazing message if you got something from this video subscribe because I'm putting out new videos all the time thanks again to the patrons that support what I do it is a huge blessing until next time God bless


Welcome to ToffeeTV, it is the player ratings for Fulham, Nill, Everton, Nill, a dissapointing game down at Craven Cottage, as I've already said in my instant match action. Terrible second half but I would take a point all day long in a game like that and I think last season we lost that game last season probably by about two or three goals. But let's get into it, Jordan Pickford in goal, some really really top saves, especially in the second half made a couple of really really fantastic saves, good presence all night, I think it's kicking a little bit got it, did that thing a little bit where it all goes a bit bit of a rush it had in the second half kicked a couple of balls out, kicked the couple into the centre middle of the parkway, gave the ball straight away but terms of like saves and stuff is some of his saves were absolutely fantastic and done exactly what he's supposed to do and again getting that consistency really important for him especially not just for us but for him for the World Cup so yeah good display I've given him an eight, really good all-round display yeah I think a lot of people will have him as man of the match because he did make some big saves especially the one early on from William which obviously was a big moment in the game early on because if you can you can see daily and you can see sort of from one of the opposition's first attacks then the computer gives you a different outlook on the game completely so good good performance from Jordan Pickford as I said I've given him an eight, Seamus Coleman yeah I mean I understand why why pick them keeping the same team and only the manager and obviously the the scout coaching that team will know what level Patterson was at and obviously maybe with the way full and play they get the ball wide obviously and get the full box really high and try and overload you get balls into the box from Mitovic he maybe may have thought with the retain that might have just been a little bit too much for Patterson and maybe thought needed a steady hand then obviously the plan would have always been given Nathan Patterson in the last half an hour but I thought Seamus not his fault necessarily but I just thought he got caught for the pace going past them and I think it was 2v1 a lot of the time Anthony Gordon not really helping him out as much I mean there was times when he was but but I don't get me wrong I don't think I think don't think things massively change when Patterson came on because obviously he's coming into a game called but I think I think Patterson obviously would be younger fitter and have a bit more pace to him fit fitter might be might be right but can cover the ground quicker could get out and maybe get to the play a little bit earlier but also there was times when Patterson jumped in so I've give Seamus six and a half and yeah as I said ideally Nathan Patterson would start the game but obviously coming back for me and you it's good that he got half an hour but we'll talk about him in a bit so give Seamus common six and a half and Michael Enco thought Michael Enco defensive is fantastic today at some of those crosses he really did make sure a lot of those crosses didn't get in the box block them at the source and I think that's that's hugely important when you play any side if the opposition because once the ball goes into the box if they win the first header which a lot of the time they did or they put pressure on you and don't allow you to head the ball away which a lot of time they did it makes it really hard and they did that a lot Mitovic is a danger if he doesn't win the ball he doesn't let someone else get a clear contact so it was about stopping it at the source and I don't make a link oh did that really really well I think that's what I think I don't think Seamus Coleman or Nathan Patterson really got a grip with that and that was so that was that was a big plus from that side yeah a lot of the time it went off the corner or a throw or sometimes we get the goal kick but it was hugely important that we we at least stop one side and I thought Mikhalenko shut that down really really well again going forward okay doesn't doesn't give us doesn't give us loads and that's something that has to be addressed going forward whether it be him or somebody else we have to have a left back that can get up and down the pitch that can defend and also can can attack like Mikhalenko did last week or we changed the stat changed the pace and plays in front of him and we do it a little bit different but I thought he was excellent Mikhalenko I'll give him an eight so well done to him the two in the middle Connor Cody I've given us seven and a half I think you know he done he done well he was again when you're playing against a big man it's important one of you deal with them and I think James Tarkowski dealt with Mitovic a lot better than Connor Cody did I think Mitovic was pulling on to pulling into the space between Connor Cody and Seamus Coleman and that's when he got most of his joy but Connor Cody defended well blocks and doesn't panic on the ball either which is really important so done well today I've given him seven and a half James Tarkowski just that little bit more I've given him an eight because I think in the air dealt with everything in the air blocks should have probably scored actually in the first half had a great opportunity to score from it from a free kick I think it was but ahead of the state of the goalie and I just thought again like last week just did that a little bit more because we needed the physical presence of him I think Connor Cody where he's great at defending this just puts people off just make sure he gets his body in the way I think James Tarkowski's a little bit of a different beast he's a little bit bigger deals with a lot more in the air and I thought today dealt with loads in the air because it was always going to be a handful they were just pumping so many balls into the box so I've given James Tarkowski an eight into midfield a just a guy broke a lot up obviously we decided to take him off but I thought it in front of the back four broke a lot up and I thought it was I thought it was notice well actually when he went off that day started to get on the ball a little bit more and getting just more central lead they seem to have more where as early did the they were just getting crosses in and listen that's just me on the high test the numbers might be different but I've given a seven I think again this and this times when he's on in possession where he's not fantastic but defensively very very sad and I thought again today consistent and really much of it should have been sent off a standard on him um yeah uh oh nana had his moments today but then again I think a little bit of what happened against Newcastle where once once a little bit too much time on the ball at times um I thought his yellow card was I mean he just stood on his foot I do think at times he's a little bit clumsy he wants to get he wants to get to the ball that so quickly that sometimes he forgets he's bigger than the opposition the opposition just seemed to fall over and he went free kicks from it it doesn't happen loads I just didn't get something that he needs to maybe have a little look at and read s and then he just needs to pick up pick the pace of the game up at times when he's on the ball um but quite solid and again he went back into that number six role he did all right um when a just a kind of guy went off so um wasn't a terrible performance from him I've given him six and a half but just just never really got going for me in the game and I did it a woe be a woe be was a woe be a couple of moments but really couldn't we couldn't thread anything together at all through a woe be which is what we've been doing in the home games but in the away games it's been very much again like Newcastle game him having the ball but not being able to thread anything between the players and in midfield we do massively lack at times that that balance that you know been able to go through the gears where other teams seem to have that balance we did really struggle to date up to in the second after to you know complete three or four passes it was really bad at times the the lack of um stability in there so I've given woe be a six good great sorry great first half was was a big threat put a great ball in for Dominic Carvenloon for really well he should have knocked into the into the back of the net really um another couple of times cut inside one he got a shot off one he didn't bit of a threat but the threat dried up second half and again when he's been relied on to be a defensive player it's it's not there it's just not there so um struggled again for me today especially that second half he just he is what he is isn't he and you don't get a level of consistency and when you do it's it's maybe just not good enough for where Everton want to go so again I think we know what we need we need one really top or native wide player I have give great six but Gordon today flattered to deceive again thought he was really poor um in possession think again final ball poor final judge poor thought he really thought he was lucky to stay on to be honest longer than great I don't he didn't really offer anything he had one really good chance in the first half that he blazed over just hit the target just hit the target it makes such a difference seeing last week what hitting the target does you know three three goals last week you know his goal himself that he got it's because Michelenko hits the target um last you know last week you get to tapping if he hits the target there it either goes in bounces off the keeper and might fall to someone you've got to hit the target in those areas you know you know great have a bit of a weak shot but at least he's testing the goalkeeper you want it to be better but um but he was poor Gordon for me again today really really poor and that's where maybe we need a change in terms of only having one one one of those two playing um you know especially in especially in the in the own away game um in an away game I think maybe you have to be a little bit bit more of a team and get them behind the ball and then break like that I think I just yeah go Gordon to me just early on the season ironically it was the away games that he was he was doing well in but I mean he did well last week at home I'll give him that but again today I don't know he just couldn't find his couldn't find any kind of I don't know any kind of sort of level um yeah very very poor I've given him a five he just has to find a level of consistency he really does and it home games are not going to be like away games you obviously going to have to change the way you play um so I'll give him a five Dominic Alvin Lewin started the game really brightly um on the front frapple you know done well to take to do the defender and pull the ball back I think he really should have scored off the cross from Gray um but again very isolated and second half found himself just on his own got a couple of times offside um on his own and that's going to be harder for a play he's just come back as well that's going to be really that's going to be a lot harder from for a play he's just come back I'm going to do a lot more work on his own um I think last week he always had plays around them and helped them join in and obviously today he was just he was just on his own the entire game as I said he got nowhere near the second half because we need um we need plays to get up with him and those two wide plays didn't and then no one from midfield got up with him and it the whole team just felt stretched in the second half and that allowed full into Dominic because we're in midfield and we're looking up and the two wide men a miles away and and then Dom's even further ahead and then we're getting picked off in midfield and we're turning the ball over a lot so I've given Dom a six um but he has to do better and he has to get plays closer to him I think that's what the changes were for obviously Ghana came on I thought he did all right Ghana in a fact he made um he made um one really key intervention in in our box where he got his where he got his little toe poke on the ball and it just took the ball away from mid to which I think it was mid to which um and that was a really key moment for us when they they they done us on our on our on our right hand side on the left hand side um to have give Canada seven for that Patterson he did all right he he got out and he he got some boxing but I thought one on one you could see was off the pace because he got done a couple of times once they broke into the box with the answer got all that patient he done him he did do him a couple of times I've given him six and a half it's great to have him back and maybe that's why the manager was reluctant to get him on it's always hard to put a player on cold especially in the defensive position when the opposition are doing well and they've got a set pattern and they're running at you and William was running at him and you know if you haven't been in the game you know I've been in the game for a while it's um it is difficult so I've given him a six and a half uh Dwight Neil um I've given him a six never really gone into the game ball a couple of times on the left but didn't really do too much with it and Malpie really wasn't on the pitch long enough so um not not our greatest performance in the world but this one we didn't we didn't get beat on a day where if we played like that like full of mad the whole mad the correct to get three points I really would so to grind the points out if it's not the worst result in the world today and we go on to less than next week and hopefully get back to winning ways hopefully um you know if we can finish if we can finish this section of the you know obviously before the break unbeaten that would be fantastic that would be fantastic so not the worst points in the world um after not a great performance so let me know your thoughts in the comments make sure to check out my intermatch tracking make sure to check out Baz's um video on to get it because he was at the match today um yeah and it will be back on Monday to do the final word over on toffee tv premiere if you want to join us for that it will be live the link is in the description and it'll be on the screen in a moment see you later