Hey everybody, so we had some more tech issues with the Going live on the new channel We're good with YouTube and everything just working out some kinks. So back on this channel for now. We'll see how things progress, but Now we're gonna get right into this first break here in just a few minutes. I'm gonna share this over to the Facebook group and to Twitter Alright, so first break is sold out. I'm gonna update our spots here and then we're gonna get this We're gonna get this going so that first break is sold out Alright, so for our other breaks we have 13 to go and heritage three box number four pick your team 11 to go and bowman jumbo four box number five pick your teams 13 to go and bowman baseball jumbo number six pick your team Bowman draft first edition that's 2021 has 16 to go. That is random team style number there. There we go Pristine is at 19 and 19. What's going on here? Oh, I've got it 16 on tier one. He's correct That's So again everybody, sorry we went live everything was looking good and Then we had some connectivity issues. So Here we are. Let's get this going Everybody Matt here for latent sports cards and we are ripping 2022 tops heritage baseball three box break number three pick your team We have Brian M. Orioles Tigers Mets Clifton K. Astros Dave K. Cubs Glen W. Angels Braves Yankees Isaac G. Pirates John H. Blue Jays Nats and Giants George M. Red Sox Matthew C. Phillies Padres A's Mitchell H. Twins Philip H. Guardians Dodgers Rangers Riley F. Rockies Royals Rob W. White Sox Russell G. Brewers Sean P. Reds D. Bax Marlins Suzanne P. Mariners Tom C. Rays and Tony K. Cardinals Alright, yeah, I guess so Riley we'll get the bugs worked out.  We're just really fortunate to have this as a backup All right, box topper is Josh Donaldson What is happening Josh Donaldson? That was twins right here we go base Heritage breaks yesterday were pretty good. So hopefully we can keep that going base. I have our first hit coming here That's A's Jersey Matt Chapman. We go open with the first hit. We will do it random at the end of the break for all Non sports Hendricks So we do have one more heritage baseball three box break pick your team style with some spots left space 12 teams left for the next heritage three box break. We have bowman baseball jumbo That's 22 bowman jumbo four box breaks two of them pick your team style. We have 11 and 13 to go on those Then and now we're gonna do a dice roll here for the duels It'll be left side one two three right side four five six for the entire break unless it's a one-of-one or a hit That is three so they will go left that goes angels Orioles there about Bruhan. All right. There's a trio off the entire break left one two center three four right five six and Then on the trios that have a big player a big picture and then to battle that big picture in the middle will be considered the middle of one two three four five six That is one So that'll go left for me Gonzalez flashbacks Clemente I'm Dolos Garcia and they look Cruz How's everybody doing this morning? There's O'Neal Cruz rookie that's freed for a parallel or something out of this box Cunha That's it for box one we go box two See something crazy here to kick off for today. That's nice Mike trout box topper We go angels. Hey, what's up Scott? Yeah, thanks, man. Appreciate it. I'm ready to Ready to be breaking in the new in my new spot, but we'll get it worked out Appreciate everybody's understanding So again on the triples that that's the center picture. So this will go left Salvador Perez But I'll be doing just baseball unless I'm filling in or you know, unless Unless there's a special circumstance trout DJ bridal judge and this is Red Sox that is team color variation Jerinder ran there you go Boston nice rookie short print Variation tops did us the biggest favor in the world by Flipping those variations and everything upside down now Definitely made it a lot more at least a lot more group break friendly. Nice color coming Roger's that is Yankees Cromer fracter John Carlo Stanton. There you go.  We do have another tops heritage up after this See how many spots we have left in that Ten teams left in the next one. Let's fill those up and then we have 10 and 12 to go in our bowman baseball four boxers Those are also pick your team style We have a bowman draft first-edition baseball three box break random team style 16 to go in that a couple tops pristine Baseball pick your team style four box breaks. We have 19 and 18 to go and then tier one baseball full cases pick your team style two of those We have 14 and 16 to go in those Two box 2021 bowman baseball jumbo random team style break and That one has 18 to go. We have a big 20 box sapphire mixer tops and bowman sapphire. There's a tonny 22 to go in that one And then 21 to go in our For 19 and 21 to go in our 2018 bowman chrome four box random team style breaks We got relics in the first two boxes. Hopefully we get a Nice auto in the last one Been a while since I pulled a ready and caught it would be okay with that base and base Here we go Hey, what's up Karen You know we all did there's zander bow guards box topper hey mark. How's it going base? Let's fill up the next one three box Number four pick your team Ten teams left in that one. Oh Nice day appreciate it. How's it going? We drop a link for that next break Ten teams to go. We have D-backs Orioles Cubs Reds Guardians Rockies Marlins A's Giants and Rangers Tony. Oh nice Dave gallon Nice and his cardinals Yadi Armalina Short print image variation very nice Yadi Congrats cardinals. I'm not really sure on that Dave It was an awesome looking card, but condition-wise like nothing jumped out at me like no issues jumped out, but I Didn't really get a good enough look. I don't think I don't have a great eye for grading anyway There's O'Neill Cruz rookie and nice cardinals Legend Otto Ted Simmons You go St. Louis great box of the cardinals with the Yadi variation and the Ted Simmons Otto Be sure to grab a spot in the next break those spots are as cheap as $10 for some of the teams We have D-backs Orioles Cubs Reds Guardians Rockies Marlins A's Giants and Rangers base About time for a wander rookie here Seen a couple O'Neill cruises. Hey, what's up Jack? Matt's favorite heritage Heck yeah You should have seen yesterday. I pulled up flip stock of Christie Matthewson Yeah, that was awesome It was yesterday. It was um, I don't know whoever had the twins Yeah, it was Washington Senators Yeah, you want this open? I'm fine. You can close it, please Thanks. Oh Nice Dodgers Joe Kelly Black parallel that's not serial number, but it has a print run of 50.  Give me just a second here. We'll do a quick recap. All right. We had Dodgers black Joe Kelly Cardinals Otto Ted Simmons Cardinals image variation Yadi Amalena Brewers Jersey Keston Hera Um Yankees chrome refractor John Carlo Stanton Red Sox image bearer. I'm sorry team color variation rookie Sharon Durand and A's Jersey Matt Chapman That'll do it for the break. Thanks everybody appreciate it. Number four is coming up next get that video uploading We have nine to go on heritage now nice Hey, thanks Dave appreciate it Check our spots here We have D-backs Orioles Marlins A's Giants and Rangers to go Six to go now, I'll be right back Yeah, it's on the website. Let me drop another link here for it D-backs Orioles Marlins A's Giants and Rangers. There you go. I'll be right back. All right You guys are in for a treat now. I've got an energy drink now. You're gonna get Mr. Excitement We're down to Orioles A's and Giants Three to go and you go three to go heritage baseball three box break number four pick your teams Orioles Orioles are 17 A's are 12 and Giants are 12 cheap spots Nice inexpensive way to get in on a group break here on a Tuesday morning. Excuse me. All right, so we have three 10 12 11 and 19 14 16 and 16 I changed in that direction, but okay, so big release day tomorrow Select football It's like a couple teams just went Yep, we're down to actually are they sold out too? Are we sold out? But yeah, we're gonna be doing clearly authentic baseball on the baseball channel on Thursday let's go in here in just a sec I Think we're sold out. Yep sold out. Thank you everybody All right, let me get my team list together here and we will get this going. Let's fill up bowman jumbo. I Need to get that one tweeted out really fast Still some good spots left in bowman jumbo four box break number five pick your teams guardians Tigers Royals Dodgers Marlins Yankees Phillies Cardinals blue jays and nationals. Hey, what's up Jake? Yeah, man. Good luck. All right Ready to go. Let's do it. Everybody Matt here for light and sports cards ripping 2022 tops heritage baseball This is three box break number four pick your team. We have Anthony W Cardinals Christopher Z Padres blue jays CJ L Yankees Dave K Cubs Farhan a Marlins Glenn W. Angels Braves Isaac G Pirates Jake G Orioles Giants Jeff in Dodgers Mets Phillies George M Red Sox Kevin a White Sox Royals Kyle B. Astros Mitchell H twins Nick in Brewers Scott G. D. Bax Reds Scott G also has Guardians Rockies Rangers and A's Suzanne P. Mariners Tom C. Rays and Vinnie H. Tigers and Nationals There we go. Good luck. See something crazy here It's a box topper Rafael Devers So one of those in the last break too. All right, let's fill up bowman jumbo four box breaks pick your team style Those are up next back-to-back four boxers. Those will be from the same case. All right There's a duel. We'll do a dice roll for the break left side one two three right side four five six That is five that goes right Again, that'll be for everything in the break except obviously the hits and one of ones there's wander Nice one raise This is Phillies Jersey Aaron. Nola. You go Philly.  There's O'Neill cruise rookie Good box here so far. All right. There's a trio. We're gonna do a dice roll for the triple-player cards We have D-vax Cubs Marlins on this one, but it'll be one two three four five six a left one two middle three four right five six on the ones where it's Where the cards are stacked and there are two on the bottom left and right and then there's one in the middle on the top It'll go the middle top will be the middle one. So Probably not the most eloquently said but thank you get the point. There's four. We'll go middle For all the non-hit trios in the break Middle here we go. Let's fill up those bowman breaks. Number five is up first 2022 bowman jumbo Four box break number five pick your team We have the Guardians Tigers Royals Dodgers Marlins Yankees Phillies Cardinals Blue Jays and Nationals still available All right. Let's see something crazy here. I saw somebody on my One of my Facebook friends is a big Cardinal fan And I actually saw him complaining about pools taking time away from the younger guys Which I get But come on. It's the wrong It's like the wrong month to say that pools is coming off of the NL player of the week. There's a few packs to go here in box one. Here we go last pack Again, let's fill up bowman jumbo for box break number five pick your team That is a chroma fracter Teosca Hernandez Blue Jays That's the six seventy three Yeah, man, it's good to see Garrett Cole's base Guardians are gone. Looks like we have Tigers Royals Dodgers Marlins Yankees Phillies Cardinals Blue Jays and Nationals still available for 2022 bowman Jumbo four box Break number five pick your team style. Yeah, I'd love to see them get 700 And I'll admit I was I mean as a Cardinal fan. I was super salty when he left, but I'm over it. He's glad to see him back wrapping up his career as a Cardinal. That is nice Flip stock for the D-backs David Peralta See the rough surface there on the front the smooth surface on the back compared to The other heritage cards again print run of five on flip stocks Yeah, I was I was afraid last year was going to be it for him Disappointed when he went to the Dodgers Nice that is Royals Throwback auto Amos Otis There you go, Casey Amos Otis sounds like a name that you would give if you're trying to give somebody a fake name But you didn't think ahead. You're just like Spitting it out on the fly. Yeah, that's awesome, Dave. All right. Let's get that bowman baseball filled up It's up next. Let's keep the day rolling. We have Tigers Royals Dodgers Marlins Yankees Phillies Cardinals Blue Jays and Nationals We had some great bowman breaks yesterday yellow Seeger all cards do ship from this break Sup beer face Let's see something crazy out of that last box Maybe a cut auto or something like that with battlefield That card sport over Carlos Stanton with Merrifield All right last box. That's cool. Well, the stamps right over the Team name but Dodgers team card from 1973 Who will now let you let's get that filled up 2022 bowman jumbo four box break number five pick your teams. We have Looks like eight teams left now the Tigers Royals Dodgers Marlins Phillies Cardinals Blue Jays and Nationals We also have um a big sapphire mixer later tops and bowman sapphire mixer 20 boxes Next month's eating this is red socks hand numbered bat relic To 99 kyle schwarber There you go, boston About time for a heritage super fracter Nice color coming that is red socks red Refractor to 573 Raphael Devers nice one boston Nice color match too. There's o'neill cruise rookie Max freed do a quick recap after this stole Eight to go on bowman baseball jumbo four box break number five pick your team Matt Duffy all the young Alex wood brandon marsh There's wander rookie Centering is not too good on that three packs to go There's otani Clemente There's a rosarana and sasugo on the back quick recap here We had rafael devers red refractor Um red socks bat route to 99 kyle schwarber Royals auto aimas otis That was a flip stock david peralta Royals Salvador Perez black the print run of 50 And Phillies jersey erinola that'll do it for the break. Thank you everybody. Appreciate it. We have bowman up next Still eight to go tigers royals dodgers Marlins Phillies cardinals blue jays and nationals Let's get these breaks filled. All right. Let me update our spots here Upload this video Still eight to go on bowman baseball jumbo. Let's fill it up It's like we have tigers royals dodgers marlins Phillies cardinals Blue jays and nationals still available So eight to go All right remaining teams have been marked down on number five I'm also marking them down on number six. Give me just a minute All right remaining teams have had the prices cut on uh bowman jumbo 2022 Four box breaks numbers five and six pick your team style We go eight to go on that first one It's like the gnats are gone. Yep down to seven tigers royals dodgers marlins Phillies Cardinals jays seven to go Looks like the cardinals are gone from number five So six to go Tigers are 89 royals are at 39 dodgers are at 109 marlins are at 224 Phillies are at 39 and blue jays are at 39 Let's get this filled and get ripping update the rest of our spots here On the tonight sheet or today's sheet 6 11 9 16 Let me refresh here Five teams. I'm sorry The six teams to go. This is 2022 bowman baseball jumbo Uh four box break number five pick your team style We have tigers royals dodgers marlins Phillies and blue jays Looks like we're down to two teams left Tigers and royals Just the tigers and royals Two teams left 2022 bowman baseball jumbo four box break number five pick your teams We have the tigers marked down to 89 and the royals marked down to 39 We're having some more spots come in for the other breaks We're at 19 now on that tops and bowman sapphire 20 box mixer Refresh here Another order. All right. We are sold out. Thank you everybody This one is sold out. I'll get that going here in just a sec Let's fill up number six Again prices have been cut on all remaining teams for the uh For number six as well We have braves guardians tigers royals dodgers marlins a's cardinals blue jays and nationals available for that one uh because the Cards for certain teams the the cards that you can get out of them Are more valuable. All right. We are ready to go Everybody matt here for latent sports cards are up in 2022 bowman baseball jumbo. This is four box break number five pick your teams We have andrew s nationals anton h twins carol g braves chris f reds pod rays eric l angels um gary m guardians glen wd backs orials Mets red sox and pirates henry c rockies jonathan h brewers joseph c fillies justin h yankies mark b tigers royals blue jays p n marlins dodgers rick e raise Uh white sox mariners tom b cubs walter b cardinals wychoid giants zach d rangers astros and a's Good luck everybody. Oh box one. Hey, what's up karen? Appreciate it. All right first up. This is Just the backwards calrally pulmonara's Getting one first bug or first pack here packed out all weird tie it up All right, so let's fill up that next one Four box uh bowman four box Sorry bowman jumbo uh four box break number six pick your teams All remaining teams in that break have had the prices slashed christian hernan does and victor and Rockies first bowman sparkle auto 299 warming burnable That is 299 green michael escado pirates about brujan and rays base auto curtis mead first bowman nice one tampa bay There you go rays be sweeney double inserts in this box convicted garret michael brady house That's aqua to 125 rickardo genovese spots for the next breaker on sale We have braves tigers royals dodgers marlins a's cardinals blue jays and nationals Three backwards geez cheater downs. Hey, you bet. Henry. How's it going? james wood That's a green paper to 399. No star. Garcia Billy's on that one spots him base paper does not shift another three inserts to mangas Wander rookie insert Again, we have braves tigers royals dodgers marlins a's cardinals blue jays and nationals available for the next break We got two packs to go box one have an auto coming here Riley green bobby witt at least he'll There's max muncie pink paper to 299 We go a's We got shorted an auto there. Hopefully we get it in the next box We're in this break anyway. There we go box two paper mangas mangas Reed debtomers and red socks rookie refractor auto to 499 connor wong card boston austin hendrick refractor Henry davis. All right, let's fill that up Next break 2022 bowman baseball jumbo four box break number six pick your teams looks like we have Nine teams left braves tigers royals dodgers marlins a's cardinals blue jays and nationals It's pigaro and victor again remaining teams in that break have the prices slashed three inserts here again cheater downs austin martin There's james wood and victor And that is reds Blue refractor first bowman auto to 150 allen serda. We go Cincinnati I'd like to find you a delacruz in here It's a pink paper to 299 cabrion a's Royals First bowman auto Dayton duny All right, two packs to do in box two mangas There's junior san quentin pink paper to 299. There's a color coming. That is 399 lava Uh, john kenzie noel Pulled a red lava auto yesterday. That was awesome. Actually, let me drop a link here for this break For this next one Still nine to go on the next break Nine teams left Here we go. Hey, what's up case? How's it going rodriguez? We're all smartinas That is christian hernandez sparkle to 299 and luke waddle Pink paper to 299 josh low and orioles Sparkle auto to 299 louise gonzalez. There you go baltimore Doing good. Just um back here on the main channel again They're working out some issues New studio nick york Ramos No lsc baseball breaks tomorrow, but uh, we will have Breaks here on the main channel Thomas, there's wander rookie insert bradley green it's christian hernandez And brewers refractor auto to 499 brandlin jarrava There you go milwaukey brandon marsh It's aqua paper to 499 riccardo genovese Eight to go on the next break eight to go We have braves tigers royals dodgers marlins a's blue jays and nationals. This is rangers bass auto dustin harris Here you go, texas Thanks to go nice color coming Francisco alvarez That is alexander vargas blue shimmer to 150 Casas and sheets All right, that's it for box three one box to go. All right, let's finish strong here There you go eight to go 2022 bowman baseball jumbo four box rake number six pick your teams that is up next cespidus motos and house What's up, boy? I do not do you marté? that's a's Sparkle auto to 299 max monzi nice one oakland Congrats a's so yep as rookie purple paper to 250 martinez That is junior san quentin purple lava to 250 nice one guardians And brady allen purple paper to 199 Let's get the next one filled up We have braves tigers royals dodgers marlins a's blue jays and nationals to go. There's calio. Watson Mr. Martin zack vine And royals first bowman auto luca tresh brady allen refractor leaders Three packs to go. I think we should still have an auto coming white socks gold refractor auto to 50 colson mott gummary Congrats white socks That is rick e with the white socks huge hit What a last box here beauty You know that one magged for sure Down to seven to go for the next break. We have braves tigers dodgers marlins a's blue jays and nationals That's a great way to close out this break so All right, I don't do it for the break. Let me move these over and we'll do a quick recap All right, we had royals luca luca tresh rangers dustin harris brewers refractor brandon jarrabah um orials Sparkle to 299 louis gonzalez royals date and duny reds blue refractor to 150 alan serda Red socks rookie refractor out of to 499 connor wong raised curtis mead Rockies sparkle to 299 warming burnable and then we had a's sparkle auto to 299 max muncie And white socks gold refractor auto 34 of 50 colson montgomery congrats white socks that'll do it for the break Thank you, everyone. Appreciate it. We have number six coming up next Six to go tigers dodgers marlins a's blue jays and nationals All right, give me just a minute. I'll be right back and we'll get this next one going. Hopefully All right, let's see where we're at Looks like we're still at six. We have tigers dodgers marlins a's blue jays and nationals That last one uploading. Did they get the internet working? Oh, okay, we can get the internet working in there. We transferred in there like close to time If not, I'm gonna overlap again. We still have some spots rolling in here for different breaks Nationals are gone Should be down to five Tigers Dodgers marlins a's and blue jays Awesome. Thank you All right, let's get that filled up the five teams to go for bulman jumbo four box number six pick your teams All right, looks like the blue jays are gone Four to go Four teams left. We have tigers dodgers marlins and a's All right four spots to go as soon as we get this filled up. We will start ripping I've also cut prices on pristine baseball Four box numbers three and four pick your team style Let me update all of our spots here four and seven 11 and 16 and then nine and 15 11 and 16 It's like the astros just sold in pristine Still four to go for bulman jumbo four box number six pick your teams. All right, let's get this filled up It's like just the tigers and dodgers left for that break Tigers and Dodgers two to go Both teams are on sale This will pick up. This is only our second day of doing the lsc baseball breaks and again for the second day We had to call an audible at the last minute and come over into the Uh, main sports breaking studio Check on our spots here It's like we have some spots coming in for pristine I'm gonna get those updated on the sheet. All right. It looks like the Dodgers are gone One spot to go So it looks like we're at nine and 15 on pristine This next one is sold out. Give me just a minute. We'll get that one going and close this door All right, good to go. Let's do this Let's fill up bowman draft baseball first edition three box number one random teams. It is up next Everybody matt here for latent sports cards ripping 2022 bowman baseball jumbo four box break number six pick your teams We have andreus a blue jays andrew s nationals anton h twins Brandon h royals chester f astros chris f reds pod rays Eric l angels gary m Guardians glen w d backs orials mets red socks and pirates Henry c rockies john s braves jonathan h brewers joseph c fillies justin h yankies mark h tigers a's and marlins uh rick e raised white socks mariners timothy h dodgers tom b cubs benny h cardinals white choy giants and zack d rangers Good luck cartaya No first time mojo. Ah, sorry man Yeah, hopefully next time. There's no lv martyre And orials first bowman auto louise gonzalez you go o's paper Montos low low It's trace weenie atomic refractor insert to 150 So next up is bowman draft first edition. That's 2021 Random team style. We have seven teams left just 49 99 a spot That is rangers first bowman refractor auto to 499 dustin harris We go texas Henry davis Brennan davis That is mark martyne him and as aqua shimmer to 125 our alvera Henderson And that is purple paper the over pigero 250 Bryce and stott rodry you guys in abram's first heritage wasn't very good second one was pretty good ryan and this is Padres first bowman auto brandon valenzuela We go san diego Nice color coming. That is yellow peat crow armstrong 75 cubs Here we go box two Let's fill up that next break 2021 bowman draft baseball first edition three box number one random team six to go on that is Pirates first bowman auto jack sewinsky We go pittsburgh. All right. I'm ready for a super here It's shellin palanco still a pretty good number of spots left in uh In our breaks today. Let's get these filled up Suleo Rodriguez aqua refractor to 125 What's up gills? Colson mott gummary first bowman green paper Nice one white socks. Just double check these chrome really fast. I just wanted to make sure I didn't miss something there In the low I was thinking oh, yeah That is d-backs first bowman auto ryan bliss Go arizona Five to go now. I'm 2021 bowman draft first edition random team style. That's up next White socks nice blue refractor auto to 150 Lenion sosa congrats chicago Montgomery the last one and now a blue lenion sosa Blue refractors on this design this year. It's so nice The last pack box two O'Neill cruise it's a rookie insert Freddie Freeman aqua paper Meyer box three Gills, what do you think if I for the lsc baseball channel? Maybe I should get like Greg jeffreys to come in and do a guest breaker spot with me. I'm torn between Greg jeffreys and the Mackie sasser ideas sosa It's egg or carol blue paper Yeti capping d-backs first bowman auto gabin conticello Go arizona Kirby torkelson arizona cruise rookie insert Break spots here Five to go for the next break We have eight and thirteen to go for our two pristine four box pick your team style breaks Like we have our redemption coming chrome prospect autograph Oscar colas white socks there you go chicago doing well Nice one chicago. O'Neill cruise rookie insert austin martin anara's Is max muncie purple lava to 250 nice one a's cunha, this is jordan lullar there's Invicta garret mitchell and nata joe That tops and bowman baseball 20 box sapphire mixer number one random teams has 19 to go last pack box three mat manning and red socks insert auto to 99 marcello meyer There you go, boston bowman in 3d autograph That's aqua to 125 Yo, swar garcia. All right one box to go See if we can do this time like we did last time save the best for the last box Still five to go on 2021 bowman draft baseball first edition three box break number one random team that is up next mingas Gabrielle Arias And mariners refractor auto to 499 adam macco You go seattle Hey brandon, how you doing? It's brian bellow or brian bellow green to 99 Jose ramos line green paper to 399 that is Philly's first bowman auto. Yo, swar garcia Should have one auto left. Let's make it a big one Would they cards are just a little sticky looks a little hard Little less than smooth flipping through on your marsh There you go All right next break is 2021 bowman draft baseball first edition Three box break number one random teams and we have five spots left. So I have trace weenie after bishop purple lava that is your rebio on helis Prado Lava three packs to go. I will do a hit recap after this Mario Hernandez There's christian her name does aqua paper. It is wander franco rookie insert And reds purple refractor ono to 250 christian roa Serrera and paper on the back and he just a sec here. We'll do that hit recap. All right.  Thank you everyone appreciate it 2021 bowman draft first edition is up next Get this one uploading It definitely could have been a much better Four boxes there the first four was excellent We are at three to go on the next break 2021 bowman draft First edition three box break number one random teams Three left on that one Get you a link All right three spots to go Get the rest of our spots updated here Eight to go on the first pristine four boxer 12 on the second Tier one nine and 15 still uh bowman jumbo 2021 bowman jumbo two box number one random teams We're at 12 Tops and bowman sapphire 20 box mixer random teams. We have 19 to go And then eight and 15 on our bowman chrome 2018 four boxers random team style. All right. I'll be right back and See where we're at. All right. Let's see where we're at You got some spots rolling in for uh thursday's clearly authentic breaks Somebody closed us out. We are sold out Awesome appreciate it. Let's get this going Any minute to set up the random Any huge hits there were some There were a couple of really nice hits in uh bowman jumbo number five pick your teams And there's a recap at the end so you can always just go check that out if you Don't want to watch the entire break Here we go, we are doing the random for 2021 bowman draft baseball first edition three box break number one uh random teams We'll randomize three times What's up paint one two and three Nathaniel l down to justin h And three times on the teams one Two and three rays down to rockies He's copying pasted over bigger and bolder. All right.  Get your link. We'll start here in just a minute. Thanks Post it over All right results are posted to the website here comes your link Oh, it's andres. Hey, sorry about that. Let me fix that I'll fix that on the random results here There we go All right, yep all set and appreciate that. Let's do it Everybody matt here for latent sports cards ripping 2021 bowman draft baseball first edition three box break uh number one random teams There's another look at those random results Let's do this all card ship Next up we have two four box breaks of 2022 tops pristine baseball. Let's get those filled up. Those are pick your team style And having some really good luck with pristine We have eight and 11 teams to go on those That's spencer torkelson foil There's a harry ford Sean burk tanner allen There's a jordan luller dale and lile There's victor mason jr Aaron's avala. That's blue to 150 There's cody morissette blue to 150 There's rilio careris zeal clark There's marcello mire louise mottos brady house There's mason black Lonnie white See some big name low numbered in here There's luller accent wolf Thoreau pomarez chi jung lu Connor norby So again, no lsc baseball breaks tomorrow baseball channel breaks In fact, I think the schedule for tomorrow is probably all select football I haven't seen for sure yet. Nice color coming. That is headbert pares Orange paper to 25 There you go brewers More nice color coming That is cj abrams Little paper to 75 And a little pocket of low numbered stuff here There's pico armstrong 75 It's a martin There's our alice martina is blue to 150 Tommy mace All right, next box Jay rod Vargas Next up we have pristine baseball four box breaks Numbers three and four Box last box. I haven't seen any foil yet. Oh, there's one We have eight and 11 teams to go on those pristine baseball breaks nine and 15 to go on our full case On our full case tier one breaks There's nick york It's our old brown our old vera Cisco Alvarez. There's ryan cusick There's marcella meyer And elmer rodrillas cruz South frillik foil It's on jalapunia and connor norby For tops pristine baseball We still have the angels available the dodgers brewers twins pirates mariners cardinals and rangers Good teams left spencer schwellenbach Be more set We're cj abrams taro primaris Jay rod Jordan lawlor austin hendrick osalace That is eithan wilson blue to 150 Billy's Nice color that is brice miller 25 of 25 for the mariners Doug niquezy 75 Cleveland There's lawler again Pedro panetta to 50 a's Michael harris to 150 he's been a nice addition to my fantasy baseball team It's a chi jung lu to 150 red socks All right last box. Good luck getting all cards shipped in this break We have pristine baseball coming up next. We have angels dodgers brewers twins pirates mariners cardinals and rangers still available Here we go. Let's finish strong Brennan davis Jordan lawlor Roddy maricio Jordi barley That's harry Ford Dante williams jj bladay the house There's jacob jacob steymets There's henry davis ma panunhas There's maximo acosta Missile urbina urbina Shealyn polanco and shealyn polanco Michael harris Aysa lacy or hassle cunha There's jordan walker benny mott guvery cooper bowman to 50 We go yanks jay allen Packs on this side are weird jockey sespedus So again pristine baseball four box number three is up next Angels dodgers brewers twins pirates mariners cardinals and rangers still available seen moeller And gunner henderson to 75 orioles That is matt micolski 75 giants Gunner might be called this week. Wow That's awesome It's indy rodriguez blue pirates Really never recap to do on this one. So um Aaron sabato blue Oh, that was the 150 for the twins nick york blue red socks last one It's read tremble on the back and that'll do it for the break. Thank you everybody. Appreciate it. We have pristine up next We'll drop another link here So eight teams left eight teams for 2022 tops pristine baseball four box break number three pick your team There you go eight teams to go all of them are on sale All right, I'll be right back. Let's get these filled up Angels are still available mariners are still available in the uh in the next break Dodgers brewers twins pirates cardinals and rangers as well. All right, let's see where we're at seven to go now Seven teams left for this next break pristine baseball four box number three pick your teams Don't have group breaks tomorrow in the baseball channel. So if we've got some spots rolling then for uh, let's see A couple of breaks here. Let me update these Six to go All right, we have Five to go now angels dodgers brewers pirates and mariners Five to go As soon as we get this filled up we will get it started Looks like the brewers are now gone Getting closer. We have Angels Dodgers pirates and mariners four to go Appreciate a lunchbox. Good luck with them. You know angels Dodgers pirates and mariners angels are marked down to 179 Dodgers down to 49 Pirates down to 109 and mariners down to 179. Hey, what's up, Ron? How's it going? Angels trout notani, obviously mariners have ichiro And griffy and I think they have ichiro griffy duel as well. All right Dodgers are gone now Three teams to go We have angels mariners and pirates my mowing guy Thinks he needs to contact me like three times a week and he always tries to call during the work day Two to go now angels and pirates Just the angels and pirates and we can get to ripping I'll be right back and see if these uh teams are gone by then Let's see where we're at a couple more spots came in for 18 bowman chrome We got closed out now. We are full sold out For this next break. Appreciate it everyone get it going here in just a sec. Let me get our team list together I'll check on how many spots in the next one Looks like 10 to go on the next one still. Let's fill that out All right That one is sold out. Let's get this going Everybody Matt here for late in sports cards ripping 2022 tops pristine baseball hobby four box break number Number three pick your team And the first one Give me just a sec here. I need to see who unknown is That's weird. All right.  All right, so we still have spots left for the next break We have braves reds angels dodgers bro's twins pirates giants mariner's and rangers Let's get that next break filled up pretty quiet chat today Everybody must be hard at work today. That's kyle seager gold to 50 That's one mariner's That's elvis andrew's purple to 99 Cardinals jersey auto paul deyong Slice of a star you go st. Louis ready freeman quimente Sario All right, let's fill up four box number four just tautis rosarana And here we go Some case cards nice start show hailed tonic gold angels One of the last two teams sold nice one angels Joe ryan rookie blue 75 joe carter base We have broers auto to 99 freddy peralta That's mariners auto jaren kalinick and a's auto matt olson Here we go with box two Braves reds angels dodgers roars twins pirates giants mariner's And rangers still available for the next four box break see what's in box two here It's joe morgan gold 50 reds And randy johnson orange to 25 d backs sprandotatis junior purple to 99 padres Mike's danino purple to 99 rays This is phillies jersey auto zack wheeler There you go philly the tony devours Rivera Yeah, that's a bad situation Soto Beltray, all right. It's a gold duke snider to 50 Dodgers That's red socks popular demand auto zander bogarts That's one boston It's brandon marsh rookie to 99 or no. I'm sorry not numbered net castellanos And phillies gold auto erinola to 50 Nice one philly and royals rookie auto jackson coar. All right box three Let's see where we're at on spots here Looks like we're still at 10 on the next break So the report i'm getting is that Everything in the new studio is good to go So after this break, i'm gonna cut the stream and be sure to jump over to the latent baseball channel Where we're supposed to be breaking Lunchbox, you better take that back Yeah, I think they could do without encasing the base cards Although I don't know some of the parallels are nice.  How's it going by us? Post it's across the room there All right, we have Tigers rookie auto matt manning That's metz rookie gold tyler mcgill Andrew Vaughn Reese Hoskins albert poohols and nationals gold auto 25 of 50 won soto congrats gnats Very nice soto gold. That's perhan a last box again after this one i'm going to cut the feed um after we do the recap and jump over to the baseball channel because Kenzie has a like a four o'clock start time today Let me check here Yeah, she starts at four So i'm going to need to be out of her way so she can start setting up But again what i got is that everything's good to go on the uh On the new channel i have a feeling we're going to be Firmly running pretty late It's the baseball break today It's just it's new you know it's going to take a little while to really catch on but And the brakes are still filling it's just a little slower. Oh, is that you with the uh with the net? You know that's a awesome net Well, let's see a super or something here Listen to me and just pull a gold soto auto and i'm winding about wanting to pull a super Let's fill up the next break there are several spots still available That'll be ripping on the uh lsc baseball channel I'd say once I cut the stream here give me about 15 minutes to go live over on that channel So the sondra's purple and yankees That's jersey auto gold geo arcella 20 of 50 Nice card yanks base and edgar martinez purple bias brian haze cunha no tani Pedro All right lasting case cards. We have ryan al tapia mixolak Cardinals auto paul gulchman. There you go cardinals That's reds orange berry larkin to 25 Cubs blue furgy jinkins nice color match And cubs pure power auto wilson kontreras That'll do it for the kate or for the break. Thank you everybody appreciate it We're going to switch over to the other channel and I'll be live Probably at about 220 to 230 Thanks everybody So let's fill up that pristine baseball Number four pick your teams We're still sitting at 10 to go on that so guys i'm going to cut this stream and uh Start setting up over on the other stream once again So i'll see you guys here in just a bit Thanks everybody


Okay, hello everyone and welcome to EsmarConf 2022 and the special session number eight, developing the synthesis community. As always, the session is being live streamed to YouTube and the individual presentations have been pre-recorded and published on YouTube as well. Subtitles have been verified and can be auto translated for those individual talks, automatic subtitles will be available shortly for this live stream. If you have questions for our presenters, you can ask them via the presenters individual tweet from the at esh hackathon Twitter account, which is also on the slide here. Presenters may have time after their talks to answer some of the questions or at the end of the session if time allows, we will endeavor to answer all questions soon after the event. We would also like to remind you of our code of conduct, which is available on our website at github.io. So now I would like to introduce you to our first presenter Wolfgang Wittbauer from Maastricht University Wolfgang Iran. All right. Just a second. All right. Here we go. My name is Wolfgang Fichtbauer from Maastricht University in the Netherlands and I will be presenting on the metadata package, a collection of meta analysis data sets for arm. So one of the nice things about published meta analysis is that they often not always but often include the full data set that was used for the analysis. So for example, down here, you see table one from the meta analysis by coders and colleagues on the effectiveness of the BCG vaccine against tuberculosis. So for each of the included studies, we have the number of participants that were vaccinated, those that were in the control group and the number of TB cases in each of these groups, based on which you can compute an effect size measure, and then conduct a meta analysis. So by going to meta analysis, we can actually build up a whole collection of meta analytic data sets. So you do not have to go back to the primary studies and extract the information from those which would be really tedious and time consuming. Somebody has already done this. And if you're lucky, you can find the data set in a table or maybe in an appendix, and then we can put it in our database. And why is this useful while for various purposes for teaching purposes for illustrating or testing methods for validating the analysis that were conducted and for sensitivity checks. So what you see here is the BCG data set as it was included in the metaphor package. So again, we have the studies, and we have the same information that was in this table one, except that instead of the number of cases and the group sizes, we have the number of cases and non cases in the treated or the vaccinated group and the same photo control group. But this is essentially the same information. So then we can compute the same effect size measure that was used in the meta analysis by coders and colleagues these risk ratios or more specifically log risk ratios. So now we have those in our data set and the corresponding variances. Then we can pass this information to a modeling function like the RMA function from the metaphor package and conduct a random effects model meta analysis. Here I use the DL, the Der Simone and Lert estimator because that is also what was used by coders and colleagues. So we get these results. I'm not going to discuss these. That's not the focus here. We can back transform the estimated average log risk ratio into an average risk ratio. And here we have this estimate. So on average vaccinated individuals have about half of the risk as the those in the control groups with a corresponding confidence interval. And then we can compare these results to what was given in the meta analysis. And we see that this matches up exactly. So we are able to reproduce the findings or the results from the coders at our meta analysis. In addition to this, we can conduct some sensitivity analysis. So again, here are the results that we just looked at, but we could maybe use the mental Hansel method, which is a fixed effects model, or use one of these binomial normal models or various patient models. For example, here are the results from the base meta package that we heard about yesterday. So we can compare these results, see how different or similar they are and essentially do a sensitivity analysis. So over the years, I've included more and more data sets like this and the metaphor package, each of these dots is a release of the metaphor package. So over time, you can see that the number of data sets keeps going up and eventually I got to around 60 data sets. At a certain point, though, the idea arose of moving these data sets into a separate data package. Why? Well, it would make it easier to add data sets even without updating the metaphor package itself and the code in the metaphor package might not change, but you might want to add some data sets. And also then it would be maybe a bit easier for others to contribute data sets, it might seem a bit more daunting to make a contribution to metaphor, instead, if there's a data package, then this is maybe a bit easier to make a contribution. So we started working on this, on this at the 2019 evidence synthesis hackathon. So together with Thomas, Emily, Daniel, Alistair and Kyle, and then eventually we released the first version on CRAN two years later. So there was a bit of a delay for various reasons, but eventually we got it out there. So here's the link to the CRAN page, the development is running over GitHub, as everybody does. You can read the documentation nicely formatted online using the package down package and the metadata package now includes 79 data sets. The data sets all have a consistent naming scheme, so they're called DUT period, then author. So the first author of the meta analysis from which the data set was extracted and the publication year. And these data sets are all documented in a consistent manner. So you have a general description of the data set, description of each variable included in the data set, details about the data set or the meta analysis, the source of the data. So that's typically a publication from which the data were extracted, maybe some other relevant references in case the data set has been used in other publications. Then the person who extracted the data, so in case you do find a discrepancy between the published data and what's in metadata, you know who to contact. And then examples illustrating the use of the data. And this may even be a full blown replication of all of the analysis conducted in the original meta analysis and then some concept terms. So what are these concept terms each data set is tagged with one or multiple of them, and they may pertain to the field or the topic of this meta analysis you see some examples here. They may also describe the outcome measure use so you have meta analysis using correlation coefficients or as we saw earlier risk ratios and maybe standardized mean differences. And then also the types of methods that were used in the the meta analysis or that can be illustrating with it illustrated with a data set. So for example cluster robust inference or multivariate models network meta analysis. There are some data sets with some outliers so this is quite interesting to look at or maybe data sets that can be used to illustrate publication bias or evidence for that. So these concept terms are really useful to find data sets that you might find interesting. But of course they need to be used consistently across all of these data sets. And if you make changes to concept terms. Well that's a bit tedious because then you have to go back through all of the existing data sets and make sure that they attacked accordingly. And under this link you can find a listing of all the concept terms that have been used so far. If you just want to see the data sets included while you can do this here with this command or you can go to this link. So let me just click on this just to show you right so you have all these data sets, and then, well, you can take a look at them. So again description the format some details the source, maybe another reference related to this who extracted the data set these concept terms and then examples, illustrating the use of this data set. You can also search for data sets the package includes the dot search function, which allows you to search based on the concept terms by default or full text search of these help files. So for example if you want to find data sets, tagged with standardized mean differences, or maybe multiple key terms arts ratios multi level, or if you want to do a full text search. Well then you just set concept equal falls, and then these had files will be searched based on the full text. What we also want and this is really one of the most important aspects here is for people to contribute their data sets to the package. So we have set up a pretty detailed workflow of how how this should go. So how can you contribute your own data set if you have done a meta analysis and you would like to include it in the metadata package. And there's some guiding principles here. So first of all we want these data sets to be named in a consistent manner. We also make a distinction between the raw data in the data set that is actually included in the package. The meta analytic data sets are often large, not so much in terms of the number of rows, but the number of columns often you extract a lot of information about the studies included in a meta analysis. And not all of these variables might be so interesting for inclusion in the data set that goes into metadata. So there's the raw data set that might have been extracted from an appendix. And then there is a data preparation script that takes this raw data set and turns it into maybe a slightly cleaned or condensed version of the data set that actually goes into the metadata package. And in addition we wrote a function that helps people to document their data set. So what this function does is creates a template for the help file. Not everybody is familiar with how to write these RD files. And this function will set up a template that you just need to complete them. So that's pretty straightforward. So once you have put together all these files, you have the raw data file, the data preparation script that creates the our data file and the help file. You can either just make a pull request via GitHub if you're familiar with the Git workflow or just send us the files and we'll be happy to include them in the package. So at this year's conference, we also were running a little hackathon along the way. And so what was the goal of this hackathon while we wanted to create a shiny app everybody's creating shiny apps these days so we also wanted to we also want a shiny app. And what does this app do or what should it do it should sort of replicate what that search does, but in a shiny way right in a slightly fancier, more interactive way. And the hackathon was also an opportunity to add some additional data sets maybe improve the documentation think about additional functionality, and I'll be happy to report on the outcomes of this hackathon at the next at the closing session. So that's it. Thank you for your attention. I want to thank my collaborators on the package all the people who have already make contributions to the metadata package. And if you have any questions comments or suggestions, I'll be happy to hear them. Thank you. Thank you Wolfgang for a really fantastic presentation and package and we do look forward to hearing about the updates during the closing session today. We are going to hold off on questions until after all the presenters have gone so don't forget to send us your questions on YouTube or by Twitter. And right now we're going to head over to our next presenter. I'm Mark Lejeunesse, who is coming to us from the University of South Florida. So, Mark, you're on. Hi. I'm Mark Lejeunesse and forgive me for the candid presentation here but I'm really going to talk about three lessons I learned about trying to get undergraduates involved in screening studies for systematic reviews or meta analysis. I've also tried to use them to help classify things for test data sets for machine learning learning projects. And I'll just flat out say that it's not a straightforward endeavor as you could imagine. But what I want to start with is what's my motivation for doing this in the first place and every year or every semester I get really really excited when I'm standing in front of an auditorium of potentially hundreds of students, thinking well, how can I leverage this situation into a scenario where I could steal all their energy to code, classify, screen, many, many, many studies for systematic reviews. And so I've tried this for about eight semesters now. I'm doing one right now for my parasitology course. And so I'm going to try to describe a little bit some of the challenges that I face when getting so many students involved in this process. And maybe most importantly how I fall flat every semester in achieving high quality screening decisions by these students. And oh boy, let's just jump into it. Let me talk a little bit about the population of screeners. In general, they are undergraduates, they can range from their first year at the university straight out of high school to seniors, they've been around for four or five years. And so you could write there you know that there's a variation in confidence and experience with science. And of course anyone who's ever tried to embark in a screening project. You really need to understand the language involved in how science is reported and summarized. And so the first challenge is undergraduates tend to vary quite a bit in their expertise or I wouldn't say interest with their confidence in making screening decisions. The range of classrooms that I've experimented with, and these are not formal experiments, these are just me getting excited each semester, trying to get students involved. There are about 40 to 200 students. And it doesn't matter the size, or the number of students involved. Really, the challenge is making sense of what they screen and what they achieve. Now the last thing I want to discuss is a the number of studies that I've attempted to a screen or classifier code with the students and I haven't been overly ambitious with this. It's usually about 300 to 2000 studies. And the reason I'm not super ambitious like when you have 200 people ready to go. It really almost makes it impossible for me to verify the decisions they make. And so if I we go super ambitious and try to screen 10,000 studies. And in terms of quality control that makes it very inefficient for me to proceed with other phases in a research synthesis project. Because I have to verify and check and make sure that everyone is on the same page in terms of making good decisions to screen to exclude include studies for systematic review. Let's begin. Here is my first lesson I learned by using hundreds of undergraduates to try to screen studies. Dual screening designs really do not make sense in this scenario. Now a typical dual screening design is you have to reviewers screen the same collection of studies, and then you use a say a CAPA statistic to assess consistency between the reviewers in how they agree or disagree with what should be included or excluded. This really breaks down when you have hundreds of students, because the paired design is not adequate enough to create high quality, repeatable decisions. And the reason for that is not all students are on the same page, or at the same level of making the screening decisions. No matter how much work I put into it to prepping them, you know getting them ready with using some sort of pico framework of trying to answer questions when they're screening the title and abstract. There's always a cohort of studies of students I mean that are just not not making effective decisions. And because we tend to approach the screening process as part of a project, we have many phases to go through in a semester. And so I really don't have much time to train them and test them over and over and over again to make sure that everyone's on the same page. So the dual screening stuff really doesn't work because what you end up happening is having to a drop entire clusters of poorly graded CAPA statistics collections of studies that were just complete inconsistency in screening decisions. Which makes the whole process again inefficient because you got to, we have to revise and review these things over and over again. And when I first started this, I would repeat that process three four times, right so what would just typically be one screening about would snowball into a second, and a third screening about just a double triple check screening decisions. So now I kind of converge on a totally different approach where I'm not evaluating groups based on consistency. I evaluate individual studies on consistency. And so what I do is I take a random sample of the students, say about 20 students in a class, and they each independently evaluate a single study. And that becomes my measure of decision consistency is kind of like the sampling experiment where I'm having many, many, many students evaluate the same title and abstract for inclusion and exclusion. And then if the statistic the consistency statistic is high, then it's probably means that there is is valid to be included or excluded. So this leads me to my second lesson. And this one is by far the one I have the most trouble with at all. And I'm going to need some time to explain this because I, it's not at all intuitive. There tends to be a trend of high consistency in what to include when screening studies right students agree very highly on what to include. What has the most variability or variation in screening decisions is on what to exclude. In an ideal scenario, right there, the probability of inclusion and exclusion should be similar. However, here in this case what we have is students really hesitate and and are challenged with making exclusion decisions. Forgive this cloud of data points but this is the, this is the nature of using undergraduates for screening decisions each axis here represents a two bouts of screening decisions on the same studies. So one week we screen 250 studies. The other week we re screened those 250 studies. So here is the is the repeatability statistic of making a decision on whether or not to include or exclude an individual study. So each point here is an individual study, and about 20 students made decisions on whether not to include or exclude that study. So based on two bouts of the same studies you can see there's a lot of noise, right, even though they're doing the same thing over again. Decisions are not highly repeatable you can fit a line through this and there is a correlation between the two. But the important thing to emphasize here is that there's high agreement want to include between two separate bouts. But there's a lot of noise associated with excluding the studies. So, you know, even though the Pico statement is very useful in making decisions on what criteria. You want to hit on when you're reading a title and abstract. It doesn't really help you much with making a decision on what to exclude, especially if you're not fully confident in understanding scientific summaries and abstract is a very unusual type of writing, right, it's concise, it's short. It tends to be way more dense in jargon sometimes because you got to squeeze in as much information as you can in a small amount of space. This actually makes it I feel like makes it more difficult for undergraduates to read and understand what's happening, especially when we're trying to do a research synthesis project. This leads me to my final challenge or lesson learned. And I this is by far the dimension that I experimented the most with it again informal experiments is a the tools will break. I mean when you have 200 students, and they're using their phones, they're using maybe their parents computer, maybe they're using computers at the library to do screening decisions. That inconsistency is makes it difficult to on my end to make sense of what they achieve. So throughout the years I've experimented with many, many different things. I started off with using R generated HTML forms that populate Google spreadsheets. But what happens with that is not all browsers are friendly or open to form fillable things. And because people are using their phones and different tools to navigate bit to browse the HTML files. This creates a lot of hiccups when assignments are due. And usually there's like a cohort of students who are having problems submitting their screening decisions, because that there's like a technology issue going on there. Other approaches I've used at PDFs.  However, once the PDFs are in the hands of students, they're using different applications to to read these things. And, and then when they save their effort. It's in a totally different format. And when it ends up in my hands. That results in a bunch of implementation issues because PDFs are by far the hardest files to crack open and mine information especially form fillable data, because depending on the application you use to save the form. If you want to populate the forms, it may be in a totally different format and ours not really set up to process variations in form fillable out decisions. So what I've converted on now and it's really just this ridiculous acrobatics is I use canvas, which is the software they use to take exams, watch lectures, all that stuff and I bamboozle it for a synthesis projects. In R what I do is I take a CSV file of abstracts and titles. I convert it to a I always forget what the file name is I got it written right here, a QTI format, which is the only way canvas accepts questions, and I convert the quizzes that they take online into screening, and I convert it into a revision technology. This is by far the most efficient way for me to do stuff, even though it's harder for me to implement. It's easy for them to do because they have to use canvas for all of their lectures. And when they're done their assignment they could forget about it and I get it in a semi convenient consistent way where I don't have to worry about differences and how they saved their outcomes or how they submitted their outcomes.  So, there's many places to go with this. And I feel like maybe the greatest inroads could be technology wise in putting things in the hands of students that are straightforward and simple to make screening coding decisions. That is independent from what they use to make decisions and how they submit their decisions. And I know there's a lot of meat resources out there to do this. But I feel like they tend to be targeted towards researchers, people with a lot of experience in making decisions. They have experience with technology. And these tools don't quite fit an undergraduate who is probably 5% committed to the course, realistic, right. So they have like five courses going at once. They have their professors forcing them to do screening decisions. I mean, what we have to do is streamline that whole process to make it as simple and straightforward as possible for them. So that's it. I like to end it at that. These are some of the lessons I learned. There's certainly more, more things that I've learned. But I'm doing a shout out here. I have many ideas on how to do this, but it'd be fantastic if I could collaborate and figure this out, crack this nut with many other people rather than me just doing this on my own. Yes, what I'm bringing forward is experience and expertise in not successfully implementing this strategy for screening studies. And so, again, there's a lot of room to grow. And I think I'll end it at that. Thank you for watching. Thanks Mark. That was a fantastic presentation and just really highlighted some of the challenges that I've even come across in my own experience, but I haven't actually nailed down to those, you know, three issues the way you have and I'm so glad that you brought it to the hackathon because this is, I think the perfect group to try to work towards those challenges so hopefully when we do the Q&A, there'll be lots of time to talk about that further. So now we're going to move on to our last presentation. I'd like to introduce David Hobby and Alexandra Banach-Brown. Both are at the Berlin Institute of Health and they'll be going next. I'm David Hobby from Camarades Berlin, based at the Quest Center of the Berlin Institute of Health. I'm going to discuss a web app that we have built for teaching meta-analysis and ARC. Camarades Berlin is a group located at the Charité Health Network in Berlin for the promotion of the benefits of systematic review and meta-analysis of animal studies. Camarades stands for collaborative approach to meta-analysis and review of animal data from experimental studies and is part of an international network of pre-clinical researchers. We do our own research and we also provide methodological support for researchers to perform robust, high quality reviews. One of the ways we do this is by Sorry everyone, it looks like we're just having a problem with the presentation on our end. It'll just be a minute while we sort that out. During the pandemic, the tech hurdles which have always been present when teaching this material have been exacerbated by the remote learning framework. Most of the participants in our workshops come from disciplines such as biomedical and medical research or epidemiology and are generally not familiar with programming languages or statistical programming software. In previous iterations of the course we would provide an R script and some CSV data files lay out the technical requirements in advance with the expectation that all required software be installed and functional on session day. However, some subset of the participants would often arrive without having read the requirements or installing R or the required packages or without a date or conflicting packages. This often turned the first hour or hours of the course into a troubleshooting session where we'd have to go around and figure out what was going on with each individual person who was having issues. In light of this issue, we had the idea to build a self-contained web app based on the Learn R package, which walks the user through the steps to recreate the analyses of a published meta-analysis of animal data in the biomedical field involving a comparison of controlled intervention studies affecting infarct volume. This publication used all of the medical and analytical methods which we would like to teach, namely random and fixed effect models, meta-regressions and visualizations of heterogeneity with study design characteristics, forest and bubble plots. The two R meta-analysis packages which we used, meta and metaphor, contain the necessary functions for performing these analyses. In order to present these analyses in a useful way, we also used two additional packages. These packages were Shiny and Learn R. Shiny is a package for the creation of interactive web apps in R, which when combined with Learn R allows R markdown documents to be converted into interactive tutorials with live code exercises. Here, for example, is one code chunk taken from section one of our app where the user is asked to examine a data set that was just loaded. When you click Run Code, the code is executed and the output is displayed in the browser. After each major section, we also have questions with quizzes to check for understanding with immediate feedback. So if you get the wrong answer, you can try again until you get it right. In section five, participants are asked to create their own code based on what they have learned from the preceding sections in order to answer a set of questions. If they get stuck, the hint button will point them in the right direction. Other features that can be implemented in Learn R and Shiny are embedded videos and interactive Shiny elements. We actually didn't use either of these in the current iteration of the app, but it could be possible in the future. Using a teaching app allows us to sidestep any software-associated problems that individual students may have so we can focus on the actual material and on the principles of meta-analysis. The app runs on a browser window, which means that from the student's perspective, it is an interactive website. The course is presented as a step-by-step tutorial with interactive code exercises which walk the student from loading the data through to performing the final analyses. At each step, there are short quizzes to check comprehension. Progress is saved as the user progresses through the app so that they can leave and come back whenever they want. All code is run on our own server and we make sure that all packages are up to date to ensure full functionality. This does, however, present its own issues on our side. The biggest issue we've had is the performance. Our courses have between 10 and 35 participants, which means an instance of R running on our server for each participant. And even with lower numbers of participants, we have had slowdowns and crashes when everyone is running analyses at once. Secondly, some participants reported that the user interface is still quite unintuitive and could be streamlined. One of the sections in particular is very long and is difficult to tell how far you are through it. We could probably break this up to make it more intuitive. We welcome other feedback about the app in general and the user experience in order to improve it. The biggest benefit and the primary goal of using this approach is that we have had zero technical issues on the participant's side since we have been using the app. The lowered barrier of entry for those students without prior experience or exposure to programming languages and we have received generally positive feedback from our students about it with the majority reporting that it was easy to use for the purposes of our tutorial. We provide access to the tutorial 24-7 on our website, which means participants can access the tutorial at any time and review the material. It is accessible to anyone else who may have interest. It was developed collaboratively via GitHub using free and open source software. And lastly, it has been shared with other camarades locations. Our colleagues at Edinburgh are also using the app in their tutorials and they are able to run it on their own server after downloading it from GitHub. So the app is not yet in its final form. The next steps are firstly to optimize server usage to improve performance when many users are running it. We have already optimized caching and we are investigating methods of parallelization. This will reduce and hopefully prevent the slowdowns and crashes that we have experienced so far. Secondly, we are going to clean up the user interface to make it easier and more intuitive to walk through. The one very long section that I mentioned earlier could easily be broken up into at least two other sections as a starting point. We don't want to remove any material. We believe it is important to present the full analysis from the published paper, but we could definitely arrange it in a more digestible way. Lastly, we would like to refine the questions and provide more in-depth feedback as to why answers are correct or incorrect. Thank you very much for your attention. I would like to thank everyone on the Kamaradis Berlin team and would like to thank our funders at the Cherite 3R department, as well as the developers of Meta, Meta4, Shiny and LearnR packages for making this project possible. And again, we welcome any feedback regarding the app or the experience and I'm looking forward to the discussion and questions coming through on Twitter. Thank you David and Alexandra for that wonderful presentation. And you know, I'm delighted to say that we have all the presenters here for live questions, so please send your questions through on Twitter. If the presenters could turn on their videos and microphones, we'd love to see your faces. I think I'm just going to get questions started for Mark, it's from Neil Hadaway. This is actually a question that I had a similar kind of question about crowdsourcing. And so the question is, should we be gamifying the screening experience for crowds that are involved in reviews, as opposed to kind of coordinators managing the projects? Do you think that would be a good solution? Exactly what I'm up to is really it's gamifying in that, you know, you're using a bunch of non-experts to make decisions on what to include, exclude. And I say maybe it's possible, like I have not succeeded with it. The idea is to get a lot of eyes on a single study. And like the dual screening, you just have two people making decisions that's totally inadequate. I mean, as far as I could tell, 20 plus kind of washes out the few screeners, the few reviewers that really have no clue what decisions they're making. And it's just, should we do this? I think if we could process, if you have the capacity to have many people screen a single study, then I think it's probably okay. The challenge is like you're wasting a lot of effort there. And that not a single study gets a lot of attention, but if you have thousands, then it's a distribution problem on making screening decisions. Yeah, yeah, great point there about resource waste, right? Wolfgang, I see your hand up and then we'll go to Alex Andrer as well. Yes, so Mark, very interesting. I really like this idea. So in meta-analysis we often worry about dependencies these days, right? Big topic. So how do you avoid that students are sort of getting together and doing the coding together and then you get dependent codes? Well, okay, so every student gets a random collection of studies. And so maybe two students at 20 students out of 100 class will get the same study. But how do you coordinate that, right? And so maybe, well, I do have an open discussion board for difficult situations and students typically vote to make a decision on whether or not to include, exclude. But yeah, you're right. So there is, there is that challenge. But as far as I know, it doesn't really happen because, again, students have like 30 titles abstracts to screen. It's a totally random collection from the sample population to screen. And so they don't have, like in a dual screening design, you have two people with the same collection. So they can coordinate. But when it's complete random sample, then coordinating for individual studies I think would be really tough for them to manage. Although, who knows. Alexandra? Thanks, Mark. I just wanted to share some experience that we've had with crowdsource meta research projects on work that that's been published. We have used free training. So reviewers that want to be involved in the project need to pass some level of training before to check comprehension of things like the Pico or or that kind of thing. A platform like learn our could be used to do that the learn our functions. Yeah, it can record how many people have passed not passed you're not able to move to the next section until you do pass it things like that. And you can also download it as an instructor you can download class results and individual student results as well. I think it's really interesting your dilemma about kind of randomly presenting articles or not for screening. And one of the software that we use randomly presents the articles for title and abstract screening. We have it set at two kind of agreed decisions. So that's whether there's three people involved or whether two people initially kind of agree already. And those just kind of get presented in random order. You could also increase it to say that, you know, five people need to agree in order to make that decision. And to that paper included or excluded playing around with. Yeah, how many screeners is required for a paper to be included.  You instead of it being like a binary include exclude. Now I have like a proportion. Like a range from, you know, minus one to one. And that's what basically this cloud is right is like 20 students on this individual study right here. On average, made a decision to include it. And so what I do is like I kind of convert this into space if you occupy this space here, then the study gets included. And if you occupy this space, the study gets excluded. And if you occupy either of these competing or complimentary spaces, the whole studies get reviewed again, which again, totally inefficient. Sounds like a really complex algorithm. We usually work with three screeners and then the agreement has to be above 66%, but with 20 plus screeners that seems like there's the even you even weirder thing that I did not talk about is the way this is done is the number of screeners assigned per study is also random. So you could have up to 10 to 50 students screen the same study. Despite the challenge I feel like I need to like collaborate and harness some of your student power because it would be fantastic to have this this body of students that are willing to do this work. That's why I get excited every semester I'm like holy smokes folks, let's do some science let's push push things forward but then you know not everyone is motivated not everyone is like committed. You know everyone's exciting at the beginning of the semester but when the semester is near ending and they have to worry about exams and all that stuff they really don't want to be spending time screening studies. That makes a lot of sense. Oh, Alexander here. Sorry, I just have another question on that point. I wonder how much. I mean I don't know how you set up your course about how much this contributes to to class credit, but something that's worked really well for us in crowdsource projects in the past is having a leaderboard.  So that works really well for us. I do that when we reach the stage of finding research articles where it ends up being just a giant competition or who could find PDFs of certain things. And if something does not get done, like a student does not submit their papers on time, then that effort gets distributed as bonus points for the class as a bounty to get things done. Yeah, I love all this stuff again but it's to me the implementation has always been the real hiccup Wolfgang I saw your hand raised as well. Yeah, I would like to address David's presentation for for the moment because I think this is also absolutely fantastic for for about a year or I've had on my to do list for metaphor create a swirl tutorial or learn a tutorial. So thank you. I can scratch one thing off my list. And really it's it's it's really absolutely fantastic. I was wondering how easy is it to take what you have set up and sort of plug in a different data set into into sort of the learner tutorial you have created. That's my first question. And the other one. I believe at the moment you're not using this in terms of grading or anything like this in terms of quizzes is really like a teaching tour, but have you also considered making this part of like a grading type of system where you can also give people quizzes that they're graded on. Okay, yeah. Yeah, regarding your first question. It shouldn't be too difficult to switch the data set it only took us a matter of days to get this up and running from the previous script that we had. Yeah, so I could see it being quite straightforward to switch it over. And I have a question actually for both Wolfgang and and David just you know in terms of this is from Neil in terms of thinking about you know the resources that that you both have provided to the community. Do you think there's a way that we could combine this work to you know really vastly scale training and evidence and this is kind of globally or how could we maybe do that. Because these are great resources right. But it's thinking about that pipeline about how to. Yeah, so I'll let either of you answer. Well, I mean if I if I may I can, I mean if you really want to sort of think about so if it's not so difficult to to sort of swap out the data set. I would almost like sort of imagine a system where the user can choose the data set out of the data sets for example a metadata and dynamically the the loaner to toll is then created using that particular data set. I mean that would take quite some effort to really like automate this process. There are other types of issues and methods that you can sort of illustrate with the different data sets will differ. But I mean that would be that would be like fantastic right like choose one of these data sets that you are really interested in learning more about right that becomes more motivating if it's a data set that you are actually really interested in. And automatically a sort of learner tutorial is created on the fly tuned to that data set. I mean that would be amazing. Yeah, that would be fantastic I see Alexander you have your. Yeah, no that would be really cool. I can definitely see. Yeah, many potential avenues for collaboration in the future creating a modular shiny app where these learner components are integrated so that you know, depending on what the effect measure is depending on what the heterogeneity. Investigation is like subgroup or meta reg those kinds of things that that could be really cool and to make it to make it more accessible. I think our app is obviously for teaching preclinical animal systematic review. So a lot of the information is is yeah specific to that. But yeah we use a published data set as an example so people can recreate the the analysis from that. On that note Wolfgang I wanted to ask if we could add a new effect measure to your metaphor package the normalized mean difference. Then we wouldn't need to hard coded ourselves and we could use it straight from your package but we can talk about more that more later. Well I'll just get in touch. I mean, happy to to hear what you have in mind. Thanks everybody I'm just double checking about any. Any questions that are coming in through the through the chat. So here I had had a question regarding use of pulling data sets into metaphor automatically if if if we would ever be able to kind of reach out like data that's published from preprint servers and kind of into this system. I don't know if that's a possibility. Maybe a question for Wolfgang but Marcus shaking his head. Yes, so you're saying that would be ideal. I don't I should not answer this. I don't think I could do that but Wolfgang might. I mean you can if you have a CSV file or whatever lying somewhere on OSF I mean you can that's one command to read it in. And, but, but if you want a data set properly in metadata you really need to document the data set right so yes that's really the the the kind of the more tedious thing that you have to really create a code book for every variable in the data set but but that's the that's the sort of formality that we really want we want well documented well described data sets. Right so it's easy to just read in some data but then you're looking at 50 variables that you have no idea what they mean. So it's a little bit of extra effort involved. And the next step from that is having these file formats for meta analysis data sets as a standard for any publishing. So yeah future future goals that there is this documentation and metadata for all data sets so that we can use the same code scripts for those dreams. So when you integrate a data set in your package you're the you're the one who's doing all the consistency and formatting right, or you get help right. I hope people supply really the data set the data preparation script that turns that into this our data file and this help file right. If even if you don't know how to write our help files there's the product that function that sets up the template, it even looks at the data set figure figures out what are the variables in the data set. Are they numeric other character variables with sets that all up for you. And then you just need to complete the description of each variable and of course the title. And you need to add the reference and so on. But I mean that's just copy paste from another example and then just adjust right I mean it's, it's not super complicated. But yeah I do or my collaborators we do want to take a look at at these submissions right so to make sure that they are well described and clear and so. But yeah, we will be happy to work with you on that if you have a data set and you just need help getting it into metadata just get in touch and and we can, we can get you started. That's a great offer and I think one of the things that that I find so helpful about your your work. I mean, I could actually see on you know someone like Mark maybe having if he had a group of students who he felt like we're a little bit more advanced, starting to use your metadata like creating a data file from that systematic structure and then using that as a teachable like okay maybe you got it wrong here but this is actually these are the elements that we're looking for to have kind of transparent and open science and this is why this is important so all these tools I see kind of fitting together to help you know develop this larger evidence synthesis. Yeah that's like a really important learning outcome is annotating data sets. And that's why for reuse. It's surprising how many things are floating out there with little or no information and you're kind of left to figure it out. And that's why this is so important to have these packages because that is has been filtered reviewed in a consistent way. I mean it's absolutely fantastic. Alex I thought I saw your hand go up and maybe I was a minute ago maybe I was wrong. I'm not sure that the questions coming in from Twitter were also being. Are there more questions from Twitter at the moment or do we have time. I don't see any here. Okay. Yeah I just wanted to ask Wolfgang and Mark if you your experiences with teaching are and I guess are packages to. To a wide range of audiences. Is there kind of help material that you feel is better for students at different stages or. Yeah how can we. I guess optimize the learner experience for either people that are new to like we want to teach meta analysis but they're new to are or they know are but they're new to meta analysis. Do you have any experience of if those groups of students maybe approach things in a different way. Good question. Yeah that's that's that's a tough one so I mean I teach a lot of meta analysis courses right but. And there I do not really assume that people have a lot of our experience so I focus on just sort of the minimal part about unnecessary to really get started with meta analysis of course having some. background with our super helpful. And that tends to be actually sort of the audience that I have so. People who really do not have a lot of our background and then just really want to learn how to to run the meta analysis with our. And yeah well there you you just have to in my experience what you have to do then is really minimize the our code right like. So one of the for example like it was with metaphor you can create forest plots but if you want that forest plot to look nice it's it's like three pages of code right I mean. I openly admit that this is how it is in metaphor right and so okay maybe so I don't focus on that because it's just going to be total. Overload on all these nitty gritty details of how to create nice figures with our right so I sort of show them the basic one and and just show them in principle yes you can customize this to not look so ugly. But yeah you have to you have to tune your teaching to like the audience that's super important. But yeah beyond that it's hard for me to say like yeah if you have people who are super experienced with our like how I would approach that. I don't know if anybody else has any ideas. I've tried to do it. Where the there's like multiple learning outcomes in the course where you start learning are. Then you learn stats. Then you do the two and then you move to meta analysis. And I feel like nobody walks away knowing anything with that maybe. Maybe they could learn maybe they don't know how to tinker with our but it's just too much all at once and if you could separate those learning outcomes I think yeah. Students motivations increase because they don't have so many giant benchmarks or hurdles to learn to achieve to understand the next level. And I had I again, I say that I totally I'm very unsuccessful in doing that and I try to keep those things isolated. So, I wish it was like this universal course you could take for you learn stats you learn are you learn meta analysis you learn reporting. But that's just a lot. That's like a multi year course. Maybe it probably would be. I don't know. I'm going to have to end the session we're out of time, but I want to thank all our panelists presenters. Thanks for a great discussion and for sharing your, your really valuable work with with us. And with the larger audience. I want to remind everyone that the session closing the conference closing today will be at 2pm GMT so it's in about a half hour so we look forward to seeing you all then and and thanks for participating.